Author Archives: jonsan21

5 things i learned @sqlpass #summit15

  1. Power BI is awesome! (I knew that going in, but I thought this needs re-iterating :))
  2. Cloud computing is yesterday, Fog computing is tomorrow.
  3. With all the investment to SQL Azure, the box version of #SQL16 will be the best ever.
  4. But if you’re not in the cloud yet, then you’re already behind (see no. 2)
  5. If you cannot pronounce “r” then you’re in trouble. Soon, everyone will speak R.

installing INGInious on Microsoft Azure

I learned about the existence of INGInious during my trip in Belgium. INGInious is an external code grader that can be integrated with edX. It is written in Python, uses docker, and runs on Linux CentOS server. I won’t go about and explain about it in details since it is quite documented here. But I want to document some notes during my effort to get it up and running on Microsoft Azure. Special thanks to Guillaume, the creator of INGInious, who helped me navigate the installation and configuration process.

Provision a Linux VM on Azure

I covered this on my previous post on installing open edX, the only difference here is that instead of selecting Ubuntu based image, this time I used OpenLogic’s Centos-based image.

Configure the server and install INGInious

  1. Connect to the server using an SSH client.
  2. Install pre-requisites. 
    sudo yum install epel-release
sudo yum install git mongodb mongodb-server docker python-pip gcc python-devel python-yaml python-ldap
sudo pip install pymongo pytidylib docker-py sh web.py docutils
sudo yum install python-flup libtidy
  3. Enable mongod and docker service and start them. 
    sudo systemctl enable mongod
sudo systemctl enable docker
sudo systemctl start mongod
sudo systemctl start docker
  4. Create docker group and add yourself to it. 
    sudo groupadd docker
usermod -aG docker <your_user_id>
  5. Clone INGInious and edit configuration file. 
    git clone https://github.com/UCL-INGI/INGInious.git
cd INGInious
cp configuration.example.yaml configuration.yaml
  6. Edit INGInious/configuration.yaml.
  7. Modify the containers. 
    containers:
    default: ingi/inginious-c-default
    sekexe: ingi/inginious-c-sekexe
    mono: ingi/inginious-c-mono
  8. Modify the docker_instances. 
    docker_instances:
  - server_url: "unix://var/run/docker.sock"
  9. Modify the plugins (this is to enable the interface with edX). 
    plugins:
  - plugin_module: frontend.plugins.edx
    courseid: "<your_INGInious_course_id>"
    page_pattern: "/edx"
  10. Pull the pre-built docker containers. 
    sudo docker pull ingi/inginious-c-default
sudo docker pull ingi/inginious-c-sekexe
sudo docker pull ingi/inginious-c-mono
  11. By this time you should be able to run INGInious on the Python web server. It will be running on localhost:8080. 
    python app_frontend.py

Install lighttpd web server

  1. Install lighttpd with fastcgi. 
    sudo yum install lighttpd lighttpd-fastcgi
  2. Add lighttpd user to docker and mongodb groups. 
    sudo usermod -aG docker lighttpd
sudo usermod -aG mongodb lighttpd
  3. Copy INGInious and give lighttpd access. 
    sudo cp -avr INGInious /var/www/INGInious
sudo chown -R lighttpd:lighttpd /var/www/INGInious

Configure lighttpd

  1. Edit /etc/lighttpd/lighttpd.conf. 
    server.document-root = "/var/www/INGInious"
  2. Edit /etc/lighttpd/modules.conf. 
    server.modules = (
        "mod_access",
        "mod_alias"
)
include "conf.d/fastcgi.conf"
  3. Edit /etc/lighttpd/conf.d/fastcgi.conf. 
    server.modules   += ( "mod_fastcgi" )
server.modules   += ( "mod_rewrite" )
fastcgi.server = ( "/app_frontend.py" =>
(( "socket" => "/tmp/fastcgi.socket",
   "bin-path" => "/var/www/INGInious/app_frontend.py",
   "max-procs" => 1,
  "bin-environment" => (
    "REAL_SCRIPT_NAME" => ""
  ),
  "check-local" => "disable"
))
)
url.rewrite-once = (
  "^/favicon.ico$" => "/static/favicon.ico",
  "^/static/(.*)$" => "/static/$1",
  "^/(.*)$" => "/app_frontend.py/$1",
)
  4. Make SELinux permissive. 
    sudo setenforce 0
  5. Edit /etc/selinux/config. 
    SELINUX=permissive
  6. Enable and start lighttpd service. 
    sudo systemctl enable lighttpd
sudo systemctl start lighttpd
  7. By this time, INGInious should run on the http port of your server.

I will cover integrating INGInious with edX in the next post.

summer is coming

I love travelling. I love travelling alone, with my wife, with friends, with family. And now, I love travelling with my kids, although I doubt they understand much about travelling yet. The big one perhaps a little bit.

I’ve travelled to most countries in Western Europe, several Scandinavians, a few in Asia and Eastern Europe, and a couple of states in the US. I love visiting historical places, doing activities, or just strolling along the old towns. I think travelling has enriched my view of the world.

stitch_AutoCollage_40_Images_4

I know a lot of Europeans take a break after university to travel. Since I started working right after my university days, I didn’t have that privilege. I only started travelling when I am already in the workforce. With limited vacation time and resource, it is important for me to plan in advance and optimize the travel period, so I could get the most out of the journey.

Turns out, I also love creating travel plans. I love researching the destination, landmarks and attractions, planning the itinerary, looking into the best possible route and timetable, and efficiently optimize the time and money to get the most out of the limited vacation days and money I have. Not only has it made my travel more efficient, it also allows me to appreciate the landmarks or destination better.

A typical travel plan should give you the list the top attractions, background story about the places, timetables for shows or guided tours, methods of commuting, etc. However, a custom travel plan should list the actual places you’d like to visit and those shows/activities you’d like to attend. For example, while I love to visit historical places and I usually tend to go to the top attractions from the typical tourist information page, I don’t intent to go inside every museum I visited. While I do not necessarily love operas, I wouldn’t miss a chance to sit on a performance in Vienna’s opera house. In addition, depending on your travel companion, you might need to tune to the common denominator in terms of places to go/things to do.

I think there’s an art to create a travel plan. You have to do a lot of research on the destination, transportation, and timetables. You will need to rank the attractions, optimize the schedule, and plan for wet weather. You will also need to cater for the need of your travel companions. I can certainly relate my enjoyment in creating travel plan to when I plan for projects in my professional life. It certainly helps me consider every angle possible: to optimize schedule and timetable, manage resources, plan for risk mitigation, and to satisfy my stakeholders.

While you can use almost anything to create a travel plan (including the old and trusty pen and paper), I’d highly recommend Microsoft OneNote (free download), hosted in OneDrive (free 15GB). You can create your travel plan once, share it easily to your travel companion, view it over the web or on multiple devices including your phone and tablet, whether online and offline, make changes and sync it instantaneously whenever you are connected to the internet.

travelplan

Alright – now it’s time for me to start planning for the summer vacation. 🙂

no risk, no reward

Recently, I have been asked to describe a time when I took a great risk and describe the outcome.

While I don’t necessarily consider myself as a risk-seeker, I would not consider myself as a risk-averse as well. I’d say I had my fair share of calculated risk-taking. I moved to Singapore when I was 17 to study in Nanyang Technological University. I have no family back in Singapore, although, I knew quite a bit of friends there and Jakarta is only an hour flight out. I’ve done some investing, in the amount of about a year salary, on a business that supposedly to be very, very safe, which in turn gone poof and lost all my investment. I moved to Copenhagen where I didn’t know a single soul before, albeit with a job offer from Microsoft in hand.

In any case, none of these experience top the decision my wife and I made 3+ years ago, which is to have our first child. I would definitely consider that to be in my top two greatest risks I had and will ever made.

Why would I rate that so high? After all, it is very normal for human to pro-create and care for another human being. It has been done many, many years before, by billions of people, so one could argue it is very normal and there is no risk involved at all.

For my wife and I, when we took the decision to try and have our own child, we have a picture of what is going to happen. We knew that what we decided to do was to bring a new life to this world. For a start, we knew that there’s a chance that we would not be able to procreate. We knew despite all the advancement of modern medicine and technology, there’s always the risk of something could go wrong with the pregnancy. There’s always the risk during the delivery. And if you think that is all, then you’re wrong. That is just the start. Once the baby is born, when the new life is brought to this world, then the journey has just started. You will have to care for him/her for the rest of your/their life. There’s so much you can (and will) do to ensure that he/she will grow up the in the best environment, with the best provision, and the best care you can give. But there’s always a risk where he/she could go astray, affected by the peer and other factors. The commitment we made is a life-long and the risks involved are endless. Some of the risk you can try to mitigate, but others you just have to accept and plan for them.

That’s why, for me, the decision to have our own child, is definitely in my top two greatest risks – the other being to decide to have our second child. 🙂

Having said all that, I would say the outcome has been tremendously rewarding. While I couldn’t take full credit for how well my 3 and 1 year old are doing right now, they have been a constant joy and light in our small family. The opportunity to be able to provide care for them, to impart skills, knowledge, and wisdom, to see them smile for little accomplishments, are something that cannot be quantified with any amount of money.

download elasticsearch error during Open edX installation

Bumped to this error a couple of times during my install of open edX on Microsoft Azure (check out the step-by-step here).

TASK: [elasticsearch | download elasticsearch] ******************************** failed: [localhost] => {“failed”: true, “item”: “”}
msg: Failed to validate the SSL certificate for download.elasticsearch.org:443. Use validate_certs=no or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible
FATAL: all hosts have already failed — aborting

Seems like quite a few folks had it as well. One way to fix it is to set validate_certs to no in the configuration/playbooks/roles/elasticsearch/tasks/main.yml file. While I can see that the PR has been merged, I do not understand why this error still occur to me even after deleting the configuration and cloning the configuration manually.

sudo rm -rf /var/tmp/configuration
cd /var/tmp
git clone -b release https://github.com/edx/configuration
cd /var/tmp/configuration
sudo pip install -r requirements.txt
cd /var/tmp/configuration/playbooks && sudo ansible-playbook -c local ./edx_sandbox.yml -i "localhost,"

In any case, I found that my copy of configuration/playbooks/roles/elasticsearch/tasks/main.yml still does not have validate_certs=no. In order to fix that, we need to edit the file in the server.

Here are some useful commands to edit a file:

  1. For editing 
    vi main.yml
  2. For inserting 
    Press i //Do required editing
  3. For exiting 
    Press Esc
  :wq //for exiting and saving
  :q! //for exiting without saving

So, to solve our issue:

  1. Run the following code 
    cd /var/tmp/configuration/playbooks/roles/elasticsearch/tasks
vi main.yml
  2. Go to line 37, right after force=no, and press i to insert.
  3. Type validate_certs=no in a new line.
    elasticsearch1
  4. Press Esc and type :wq.
  5. Run the following command to install open edX again. 
    cd /var/tmp/configuration/playbooks && sudo ansible-playbook -c local ./edx_sandbox.yml -i "localhost,"
  6. The download elasticsearch task should not throw an error now 🙂

installing Open edX on Microsoft Azure

Few months back, my boss installed Open edX on Microsoft Azure and document his notes here. Using his notes, I managed to setup my own copy of Open edX on my Azure account and avoid most of the pitfalls (Thanks Tim!). While the whole process should be quite straightforward, it might not be if you are not familiar with Linux, Azure, SSL etc, so I decided to create a step-by-step procedure here.

What you need

  1. Azure account. This is where we host the Linux VM (we are using IaaS in this method) for the Open edX server. Sign up for trial here, if you don’t have one already.
  2. SSH client. This is used to terminal into the server. We’ll use PuTTY (putty.exe, download here).
  3. Private/public key pair. Since we’ll be using a password less VM, this is used to authenticate with the server. We’ll use openssl.exe to create a private key (openssh key) and a certificate (.pem). Openssl.exe is installed as part of Git (download here). Otherwise look at this article on how to get openssl on Windows. We’ll also use PuTTY Key Generator (puttygen.exe, download here) to create a PPK for PuTTY from the openssh key.

Once you have all you need, we’ll start the process by creating the public/private keypair to be used to authenticate to the server.

Create a public/private keypair

  1. Create a certificate with open ssl
  2. Open Command Prompt with Admin privilege.
  3. In the Command Prompt, type: 
    "\Program Files (x86)\Git\bin\openssl.exe" req -x509 -nodes -days 365 -newkey rsa:2048 -keyout myPrivateKey.key -out myCert.pem -config "c:\Program Files (x86)\Git\ssl\openssl.cnf"

    openedX1

  4. Answer the questions that are asked.
  5. It would have created two files: myPrivateKey.key and myCert.pem. More documentation on this here.
  6. Translate the private key created earlier to an RSA private key that puttygen can understand.
  7. In the Command Prompt, type: 
    "C:\Program Files (x86)\Git\bin\openssl.exe" rsa -in myPrivateKey.key -out myPrivateRSAkey.rsakey
  8. The command above should produce a new private key called myPrivateRSAkey.rsakey.
  9. Run puttygen.exe
  10. Click Load.
  11. Find your private key, which we named myPrivateRSAkey.rsakey above. You will need to change the file filter to show All Files (*.*)
  12. Click Open and click OK.
  13. Click Save Private Key and save the file as MyPrivateKey.ppk. More documentation on this here.

Provision a Linux VM on Azure

  1. Open the azure portal at https://manage.windowsazure.com and log on with your credentials
  2. Click New, select Compute, select Virtual Machine, and click From Gallery.
  3. Select Ubuntu, select Ubuntu Server 12.04 LTS, and click the next arrow. openedX2
  4. Name the Virtual Machine, select at least A2 for the size, and browse the certificate file we created earlier (myCert.pem). openedX3
  5. Ensure the Provide A Password checkbox is cleared and click the next arrow.
  6. Name your Cloud Service DNS, select your region, open the port 80 and 18010 for edX portal and edX studio respectively, and click the next arrow. openedX4
  7. Click the complete arrow and wait until the VM is completely provisioned and running.

Configure the server

  1. Run putty.exe.
  2. Fill in the host name using the Cloud Service DNS name. openedX5
  3. Click Connection, click SSH and click Auth.
  4. Browse to the private key (MyPrivateKey.ppk) and click Open. openedX6 More documentation on this here.
  5. In the terminal that opens, log on as the user created for the server.
  6. Type the following command in the terminal to update Ubuntu packages: 
    sudo apt-get update -y
sudo apt-get upgrade -y
  7. Type the following command to reboot the server: 
    sudo reboot

Install Open edX

  1. Reconnect to the server using putty.exe.
  2. Create a screen session by typing the following command in the terminal
  3. screen –d -RR
  4. Run the one step installation script
  5. The script requires that the running user can run commands as root via sudo. 
    wget https://raw.githubusercontent.com/edx/configuration/master/util/install/vagrant.sh -O - | bash
  6. If you want to install a named release, such as Birch, just set the OPENEDX_RELEASE variable before running the script. For example: 
    export OPENEDX_RELEASE=named-release/birch
wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/install/vagrant.sh -O - | bash
  7. More documentation on this here.
  8. Congratulations! Now you have the edX platform running on port 80 (http) and the edX studio on the port 18010. openedX7
  9. See edX-Managing-the-Full-Stack for how to manage and update the server once it is running.

NOTE: On my 2 counts of open edX installation, I bumped to this error.

TASK: [elasticsearch | download elasticsearch] ********************************
 failed: [localhost] => {"failed": true, "item": ""}
 msg: Failed to validate the SSL certificate for download.elasticsearch.org:443. Use validate_certs=no or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible
 FATAL: all hosts have already failed -- aborting

While the edX portal and studio seemed to be working, I did manage to install the third time without error with a little workaround. It is documented in the next post.

a note from the past

Recently I’ve found an old note of mine, and I thought I’ll archive it here..

Snow.. and its aftermath

Three years ago when I started my journey in this half yearly cold country, I was excited to experience my first snow. When I talked about this to a friend of mine who had lived in the country for more than two years by then, he mentioned that he wasn’t looking so much to snow anymore, because how slippery and dirty the road after the snow. Not to mention the cold and chill of the weather and wind.

 So that year experienced my first snow. Three years after, i have had the chance to play a bit more with the snow in the winter and even made my snow boy (too small to be called snow man :)) Now I wonder whether I’ve shifted from the guy who appreciates and looks forward to snow, or I’m now in the same state as my friend three years ago.

 I think we as human easily changed. One day we are looking forward to something and grateful when we have them, and easily another day we think about the same thing in totally different perspective. That thing that we used to look forward to, reduced to a mere unwanted-ness and complaint. Thank God it’s not the case with our loving father in heaven. He constantly and persistently look to us with his loving kindness :).

Dec 12, 2010.

life without limits

??????????????????????Nick Vujicic is an evangelist, a president and CEO of a non-profit organization. Unlike most people, Nick has no arms nor legs. Nick will be in Copenhagen, this coming Monday – and my wife will be at Farum arena to hear his sharing, while I will babysit our 8 months old baby girl. I’ve read about Nick and his story a while ago. Even so, hearing about him again reminded me about God’s purpose in our life.

Today’s devotional from Our Daily Bread ponders about Psalm 139. David wrote how God is in control and oversaw our creation, even in our mother’s womb (v13-14). So how come Nick was born without arms and legs? Did God forget to put limbs for Nick? Where was He when Nick were in his mother’s womb? How can He think that Nick is wonderfully made?

31 years ago, it must have been very hard for Nick’s parents to see their son born without limbs. Throughout the years, I’m sure things haven’t been as easy for him compare to those who has four limbs. But we can all see now how God works through Nick, and how he is used to reach out to people in ways that we cannot imagine.

Sometimes when we are down, we think that our problem is too big and our burden is too heavy. But the next time you are faced with adversaries, remember Jesus’ crucifixion on the cross. If that’s too abstract and far away, remember Nick Vujicic. I am sure what ever you faced is nothing compare what Jesus did on the cross.

measure of a man

??????????????????????What is the measure of a man? No, I was not thinking about either Clay Aiken’s or Elton John’s song. So what is the measure of a man?

I think most of the time, we measure something by comparing it to something else. In the academic world, I guess the common way is to compare the degree someone has. Bachelor? Master? Ph.D? Or maybe how many papers he has published? In the corporate world, maybe the position in the company? role and responsibility? how much he makes in a month? In parenthood, maybe how many kids you have? your kids achievements? Even those who works in full time ministry are not immune to this either. If you were an evangelist/preacher, would you measure yourself by how many have you evangelized/preached to? how big your congregation?

When we are measuring something, the most important is the standard of measure. For example, a meter. From 1795, people has different definition for what is one meter. And it keeps changing – maybe in 2020, we will have another definition of a meter. Who knows?

Do not be bothered when someone trying to measure you by whatever standard they are using. Humans have their standard, and human standard keeps changing. The only thing that does not change is our God. He has His standard, and we should always measure our self to His – the unchanging One. Today’s devotional from Our Daily Bread ponders about the life of Job. Job had all good things happening for him, before everything was taken from him. And before he was restored, three of his friends that tried to “comfort” him, measured him with their standard and condemned him. Throughout all, Job did not conform to his friends measurement and despise God, but insisted that he was righteous (Job 32:1). Indeed Job did fell to another extreme, that is justifying himself instead of glorifying God. But in the end he repented and accepted God’s measurement of him (Job 38 -41, Job 42: 8).

So the next time someone measures you with whatever standard, ask yourself – have I measured it with God’s standard?

longing to come home

????????????????????????????“Anyone who has been deprived of home, even for a short time, will understand the comfort Jesus’ disciples drew from His words when He promised that an eternal home awaited them (John 14:2).” The text comes from today’s devotional from Our Daily Bread.

When I was a bachelor, I don’t think I was ever too excited to come home from work. When I got married, I loved to come home to my wife. Even so when my daughter was born. Now the longing to come home from work is even greater.

I guess I have never thought about John 14:2 in that way. I have never really ponder about the feeling of longing to come to the eternal home . To where Jesus has prepared for us. I have a great life here on earth, blessed tremendously, abundantly with love and other things.

I pray that while I enjoy God and His blessings on this earth, I will not let all these hinder me to see and remember what beyond this life, and deprive me for the longing to come home to our eternal home, just like the first disciples.