Category Archives: Tech

5 things i learned @sqlpass #summit15

  1. Power BI is awesome! (I knew that going in, but I thought this needs re-iterating :))
  2. Cloud computing is yesterday, Fog computing is tomorrow.
  3. With all the investment to SQL Azure, the box version of #SQL16 will be the best ever.
  4. But if you’re not in the cloud yet, then you’re already behind (see no. 2)
  5. If you cannot pronounce “r” then you’re in trouble. Soon, everyone will speak R.

installing INGInious on Microsoft Azure

I learned about the existence of INGInious during my trip in Belgium. INGInious is an external code grader that can be integrated with edX. It is written in Python, uses docker, and runs on Linux CentOS server. I won’t go about and explain about it in details since it is quite documented here. But I want to document some notes during my effort to get it up and running on Microsoft Azure. Special thanks to Guillaume, the creator of INGInious, who helped me navigate the installation and configuration process.

Provision a Linux VM on Azure

I covered this on my previous post on installing open edX, the only difference here is that instead of selecting Ubuntu based image, this time I used OpenLogic’s Centos-based image.

Configure the server and install INGInious

  1. Connect to the server using an SSH client.
  2. Install pre-requisites. 
    sudo yum install epel-release
sudo yum install git mongodb mongodb-server docker python-pip gcc python-devel python-yaml python-ldap
sudo pip install pymongo pytidylib docker-py sh web.py docutils
sudo yum install python-flup libtidy
  3. Enable mongod and docker service and start them. 
    sudo systemctl enable mongod
sudo systemctl enable docker
sudo systemctl start mongod
sudo systemctl start docker
  4. Create docker group and add yourself to it. 
    sudo groupadd docker
usermod -aG docker <your_user_id>
  5. Clone INGInious and edit configuration file. 
    git clone https://github.com/UCL-INGI/INGInious.git
cd INGInious
cp configuration.example.yaml configuration.yaml
  6. Edit INGInious/configuration.yaml.
  7. Modify the containers. 
    containers:
    default: ingi/inginious-c-default
    sekexe: ingi/inginious-c-sekexe
    mono: ingi/inginious-c-mono
  8. Modify the docker_instances. 
    docker_instances:
  - server_url: "unix://var/run/docker.sock"
  9. Modify the plugins (this is to enable the interface with edX). 
    plugins:
  - plugin_module: frontend.plugins.edx
    courseid: "<your_INGInious_course_id>"
    page_pattern: "/edx"
  10. Pull the pre-built docker containers. 
    sudo docker pull ingi/inginious-c-default
sudo docker pull ingi/inginious-c-sekexe
sudo docker pull ingi/inginious-c-mono
  11. By this time you should be able to run INGInious on the Python web server. It will be running on localhost:8080. 
    python app_frontend.py

Install lighttpd web server

  1. Install lighttpd with fastcgi. 
    sudo yum install lighttpd lighttpd-fastcgi
  2. Add lighttpd user to docker and mongodb groups. 
    sudo usermod -aG docker lighttpd
sudo usermod -aG mongodb lighttpd
  3. Copy INGInious and give lighttpd access. 
    sudo cp -avr INGInious /var/www/INGInious
sudo chown -R lighttpd:lighttpd /var/www/INGInious

Configure lighttpd

  1. Edit /etc/lighttpd/lighttpd.conf. 
    server.document-root = "/var/www/INGInious"
  2. Edit /etc/lighttpd/modules.conf. 
    server.modules = (
        "mod_access",
        "mod_alias"
)
include "conf.d/fastcgi.conf"
  3. Edit /etc/lighttpd/conf.d/fastcgi.conf. 
    server.modules   += ( "mod_fastcgi" )
server.modules   += ( "mod_rewrite" )
fastcgi.server = ( "/app_frontend.py" =>
(( "socket" => "/tmp/fastcgi.socket",
   "bin-path" => "/var/www/INGInious/app_frontend.py",
   "max-procs" => 1,
  "bin-environment" => (
    "REAL_SCRIPT_NAME" => ""
  ),
  "check-local" => "disable"
))
)
url.rewrite-once = (
  "^/favicon.ico$" => "/static/favicon.ico",
  "^/static/(.*)$" => "/static/$1",
  "^/(.*)$" => "/app_frontend.py/$1",
)
  4. Make SELinux permissive. 
    sudo setenforce 0
  5. Edit /etc/selinux/config. 
    SELINUX=permissive
  6. Enable and start lighttpd service. 
    sudo systemctl enable lighttpd
sudo systemctl start lighttpd
  7. By this time, INGInious should run on the http port of your server.

I will cover integrating INGInious with edX in the next post.

download elasticsearch error during Open edX installation

Bumped to this error a couple of times during my install of open edX on Microsoft Azure (check out the step-by-step here).

TASK: [elasticsearch | download elasticsearch] ******************************** failed: [localhost] => {“failed”: true, “item”: “”}
msg: Failed to validate the SSL certificate for download.elasticsearch.org:443. Use validate_certs=no or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible
FATAL: all hosts have already failed — aborting

Seems like quite a few folks had it as well. One way to fix it is to set validate_certs to no in the configuration/playbooks/roles/elasticsearch/tasks/main.yml file. While I can see that the PR has been merged, I do not understand why this error still occur to me even after deleting the configuration and cloning the configuration manually.

sudo rm -rf /var/tmp/configuration
cd /var/tmp
git clone -b release https://github.com/edx/configuration
cd /var/tmp/configuration
sudo pip install -r requirements.txt
cd /var/tmp/configuration/playbooks && sudo ansible-playbook -c local ./edx_sandbox.yml -i "localhost,"

In any case, I found that my copy of configuration/playbooks/roles/elasticsearch/tasks/main.yml still does not have validate_certs=no. In order to fix that, we need to edit the file in the server.

Here are some useful commands to edit a file:

  1. For editing 
    vi main.yml
  2. For inserting 
    Press i //Do required editing
  3. For exiting 
    Press Esc
  :wq //for exiting and saving
  :q! //for exiting without saving

So, to solve our issue:

  1. Run the following code 
    cd /var/tmp/configuration/playbooks/roles/elasticsearch/tasks
vi main.yml
  2. Go to line 37, right after force=no, and press i to insert.
  3. Type validate_certs=no in a new line.
    elasticsearch1
  4. Press Esc and type :wq.
  5. Run the following command to install open edX again. 
    cd /var/tmp/configuration/playbooks && sudo ansible-playbook -c local ./edx_sandbox.yml -i "localhost,"
  6. The download elasticsearch task should not throw an error now 🙂

installing Open edX on Microsoft Azure

Few months back, my boss installed Open edX on Microsoft Azure and document his notes here. Using his notes, I managed to setup my own copy of Open edX on my Azure account and avoid most of the pitfalls (Thanks Tim!). While the whole process should be quite straightforward, it might not be if you are not familiar with Linux, Azure, SSL etc, so I decided to create a step-by-step procedure here.

What you need

  1. Azure account. This is where we host the Linux VM (we are using IaaS in this method) for the Open edX server. Sign up for trial here, if you don’t have one already.
  2. SSH client. This is used to terminal into the server. We’ll use PuTTY (putty.exe, download here).
  3. Private/public key pair. Since we’ll be using a password less VM, this is used to authenticate with the server. We’ll use openssl.exe to create a private key (openssh key) and a certificate (.pem). Openssl.exe is installed as part of Git (download here). Otherwise look at this article on how to get openssl on Windows. We’ll also use PuTTY Key Generator (puttygen.exe, download here) to create a PPK for PuTTY from the openssh key.

Once you have all you need, we’ll start the process by creating the public/private keypair to be used to authenticate to the server.

Create a public/private keypair

  1. Create a certificate with open ssl
  2. Open Command Prompt with Admin privilege.
  3. In the Command Prompt, type: 
    "\Program Files (x86)\Git\bin\openssl.exe" req -x509 -nodes -days 365 -newkey rsa:2048 -keyout myPrivateKey.key -out myCert.pem -config "c:\Program Files (x86)\Git\ssl\openssl.cnf"

    openedX1

  4. Answer the questions that are asked.
  5. It would have created two files: myPrivateKey.key and myCert.pem. More documentation on this here.
  6. Translate the private key created earlier to an RSA private key that puttygen can understand.
  7. In the Command Prompt, type: 
    "C:\Program Files (x86)\Git\bin\openssl.exe" rsa -in myPrivateKey.key -out myPrivateRSAkey.rsakey
  8. The command above should produce a new private key called myPrivateRSAkey.rsakey.
  9. Run puttygen.exe
  10. Click Load.
  11. Find your private key, which we named myPrivateRSAkey.rsakey above. You will need to change the file filter to show All Files (*.*)
  12. Click Open and click OK.
  13. Click Save Private Key and save the file as MyPrivateKey.ppk. More documentation on this here.

Provision a Linux VM on Azure

  1. Open the azure portal at https://manage.windowsazure.com and log on with your credentials
  2. Click New, select Compute, select Virtual Machine, and click From Gallery.
  3. Select Ubuntu, select Ubuntu Server 12.04 LTS, and click the next arrow. openedX2
  4. Name the Virtual Machine, select at least A2 for the size, and browse the certificate file we created earlier (myCert.pem). openedX3
  5. Ensure the Provide A Password checkbox is cleared and click the next arrow.
  6. Name your Cloud Service DNS, select your region, open the port 80 and 18010 for edX portal and edX studio respectively, and click the next arrow. openedX4
  7. Click the complete arrow and wait until the VM is completely provisioned and running.

Configure the server

  1. Run putty.exe.
  2. Fill in the host name using the Cloud Service DNS name. openedX5
  3. Click Connection, click SSH and click Auth.
  4. Browse to the private key (MyPrivateKey.ppk) and click Open. openedX6 More documentation on this here.
  5. In the terminal that opens, log on as the user created for the server.
  6. Type the following command in the terminal to update Ubuntu packages: 
    sudo apt-get update -y
sudo apt-get upgrade -y
  7. Type the following command to reboot the server: 
    sudo reboot

Install Open edX

  1. Reconnect to the server using putty.exe.
  2. Create a screen session by typing the following command in the terminal
  3. screen –d -RR
  4. Run the one step installation script
  5. The script requires that the running user can run commands as root via sudo. 
    wget https://raw.githubusercontent.com/edx/configuration/master/util/install/vagrant.sh -O - | bash
  6. If you want to install a named release, such as Birch, just set the OPENEDX_RELEASE variable before running the script. For example: 
    export OPENEDX_RELEASE=named-release/birch
wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/install/vagrant.sh -O - | bash
  7. More documentation on this here.
  8. Congratulations! Now you have the edX platform running on port 80 (http) and the edX studio on the port 18010. openedX7
  9. See edX-Managing-the-Full-Stack for how to manage and update the server once it is running.

NOTE: On my 2 counts of open edX installation, I bumped to this error.

TASK: [elasticsearch | download elasticsearch] ********************************
 failed: [localhost] => {"failed": true, "item": ""}
 msg: Failed to validate the SSL certificate for download.elasticsearch.org:443. Use validate_certs=no or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible
 FATAL: all hosts have already failed -- aborting

While the edX portal and studio seemed to be working, I did manage to install the third time without error with a little workaround. It is documented in the next post.