ADAO provides apps online accessible through apps.adao.tech. These apps require a server running behind. This blog post will explain quickly (i.e. with references rather than explanations) the process to set up this server and finally the web app.
Once it is known that you will need to deploy your applications online, you will need a few things. Note that there are a million ways to set up the infrastructure. This is only one that uses specialized tools based on ADAO's objective and experience.
Domain
Domain name
Domain registrar (e.g. Google)
Server (e.g. Digital ocean)
App (e.g. Streamlit python)
Map domain name to server (Configuration with domain registrar)
Map the Web Server to the App (Configuration inside the server)
Secure with https (Configuration with Let's encrypt)
Domain name
There are many providers of domain names (just google get a domain). We will use Google. Go to: https://domains.google.com/registrar/ and follow the prompts to buy a domain. This part is quite straight forward, but a few considerations:
The domain is composed of several parts, check Wikipedia: https://en.wikipedia.org/wiki/Domain_name.
You will choose among the pre-defied top level domains such as ".com" and ".page".
You will choose a name such as "adao", "google". If someone already purchased that domain inside the top domain you may need to negotiate with them, change your top level domain or change the name.
Once the process is finished you would have purchased the domain name (say: example.com or adao.tech). You will pay a yearly fee for the domain registration, depending on the name. Therefore, purchasing a domain name is more like renting than purchasing.
Domain Registrar
The domain is controlled through a domain registrar (https://en.wikipedia.org/wiki/Domain_name_registrar). I have not noticed a difference between difference domain registrars and suggest not to change it unless you have a specific reason.
The application or website will need a computer. You have many options (Google cloud platform, Azure, AWS, Digital Ocean, Linode, etc.) I chose digital ocean because of its simplicity in pricing (fixed cost, limited resources). Check YouTube for comparisons of prices, options, etc. I will go over digital ocean, but it is more or less the same if you actually get a server online.
Note: You can set up the server in your house, which can be cheaper, but you will have to deal with the configuration of your internet service provider.
Digital ocean
You will need to get a droplet from Digital Ocean. I recommend the smallest droplet, scale only when you need it. Then you can follow the digital ocean tutorial: https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-20-04. Their tutorials are very well written and take you step by step. Once you are done, you should have an online server to which you can access with ssh.
You can create your application in which ever programming language you want. Most of the web runs JavaScript, but here we will focus on data science apps, and we will use Streamlit a python library. You need to create your application and then run it, for example (assuming you are running under Ubuntu):
python3 -m streamlit run app.py --server.port 8501
You can read the options of Streamlit here https://docs.streamlit.io/en/stable/streamlit_configuration.html. The option above, makes the server run in port 8501, this will be important later on. See the Wikipedia article for more information on ports https://en.wikipedia.org/wiki/Port_(computer_networking)
We will return to these options when mapping the server to the application.
Now you would have three independent parts 1) a domain name, 2) a server and 3) an app. At the moment, there is no relation between them. In the next three points we will combine them to create an application that can be run from a browser.
The domain name (e.g. example.com) needs to be mapped to the server so that when you type the address it is re-directed to your server. This is done through your domain registrar (in our case Google). Then Google will make sure the name and the IP address are mapped. See this Computerphile video to know more (How DNS works).
IP Address
First, find the IP address of the server (https://en.wikipedia.org/wiki/IP_address). You can easily find it on the main dashboard of Digital Ocean.
A Records
A records, as in "A" records, map a name to an IP address. Check your domain registrar and map the name to the IP address. The A record will require a name and an IP address. The name will correspond to what goes before the domain name. For example if you choose the name "apps" and your domain name is "example.com" then the A name will map apps.example.com to the IP address. This process takes between 15 minutes to 1 hour and when it is done, there will not be any software to reply to you, as this is not set up yet.
Note: You can however access your Streamlit application running on port 8501 on your server by specifying the port: "apps.example.com:8501". Note that if you have followed this tutorial your Streamlit app is not running yet on the server.
Web Server (nginx)
You need to install a web server (https://en.wikipedia.org/wiki/Web_server) on your server to handle the web traffic. We will use nginx (https://en.wikipedia.org/wiki/Nginx) but you can also use Apache or others. You can follow the instructions on Digital Ocean to set up nginx: https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-20-04. This will take you through installing nginx, fixing the firewall and guiding you though nginx's basic configuration. Once nginx is up and running you can check it by going to "apps.example.com" (see the example above), and the default message from nginx should welcome you.
Why install nginx if Streamlit can service the router?
Web traffic goes on port 80 for http (https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) and port 403 for https (https://en.wikipedia.org/wiki/HTTPS) while Streamlit runs on 8501 (or other port). Ports have special uses, and many are restricted such as 80 and 403. That is why Streamlit does not run on these ports but on 8505.
You need to configure nginx to re-direct the requests from your domain to your app, essentially mapping an address to a port. In our example, we have set up "apps.example.com" to point to the IP address of our server. This will use by default port 80. We would like a person that goes to a specific address in our domain say "apps.example.com/demo/" to be re-directed to our demo application in Streamlit.
There are three steps involved in mapping the Web Server to Streamlit: 1) set up a reverse proxy, 2) run Streamlit with specific configurations and 3) modify the maximum amount of data that nginx can transmit (by default is 1 MB).
Set up a reverse proxy
In nginx, you will need to modify your configuration. The configuration file should be the following: (assuming you have followed digital ocean's tutorials):
/etc/nginx/sites-available/apps.example.com
The configuration should be as follows:
server {
root /var/www/apps.example.com/html;
index index.html index.htm index.nginx-debian.html;
server_name apps.example.com;
location /demo {
proxy_pass "http://127.0.0.1:8501";
rewrite ^/(.*) /$1 break;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
}
I have made in bold the parts that you may want to modify depending on the specific configuration. You should read the documentation of nginx to fully understand the options in the configuration file above (https://nginx.org/en/docs/).
Run Streamlit with specific configuration
The options to run Streamlit look as follows:
python3 -m streamlit run app.py --server.enableCORS=false --server.enableXsrfProtection=false --server.baseUrlPath="/demo"
This will ensure your app is correctly mapped with nginx (https://docs.streamlit.io/en/stable/troubleshooting/not-loading.html#symptom-2-the-app-says-please-wait-forever). More information to set up this configuration can be found here: https://discuss.streamlit.io/t/how-to-use-streamlit-with-nginx/378.
Now you should be able to go to your address: "apps.example.com/demo/" and see the application run by Streamlit.
You may see next to your address that the connection is not secure. The reason is that the configuration is using http which transfer the data in plain text. I recommend these videos from Computerphile to know more: Secure Web Browsing and Transport Layer Security (TLS). You should now understand that any application online should use a secure communication out of respect. You can follow this tutorial from Digital Ocean to set up https: https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-20-04.
