Building a live-update dashboard

Recently I've finished my internship, in this post I'd like to share my experience with this internship.

Because my school allowed me to finish my education quicker (1,5 years instead of 3), I needed to find a company for my internship, quick. So I emailed a couple of companies, some never responded, some were full, some were not interested, and some were indeed interested.

After exchanging some e-mails with Exonet it came to an appointment at their office. We talked about the techniques I'd be using, what experience I had with programming and what I'd like to do as an assignment.

The idea they eventually came up with was a dashboard which fetches and processes information from GitHub and Jira and displays them so that with a glance of an eye it's visible who's working on what, and for how long. We'd eventually be using Laravel (or Lumen, we were not sure at the time), React and SocketIO to form the front-and-backend of the dashboard. To plan the development of the dashboard I'd take part in the daily stand-ups and regular sprint-meetings, following the SCRUM principle.

Nearly everything I've explained above was new for me. Whilst I've been working with GitHub for a while, and have used Jira sometime ago, everything else was new. I had never worked with either Laravel, React or SocketIO. Naturally this would make my internship even more interesting.

First step

The first step was to fetch the data from GitHub and Jira and display this in a very basic layout. Because the layout was not meant to be used as the eventual dashboard the techniques used did not really matter and so we went for jQuery and Mustache. Together they form a very easy way to display JSON on a HTML page.

We chose to write the entire backend in Laravel, a PHP framework. After using CakePHP for earlier projects at Exonet internally they've chosen Laravel as their primary language for their new API backend so it made perfect sense to go with Laravel.

After a few days I had a very basic Laravel project capable of retrieving pull requests from GitHub and displaying them in a very basic layout.

First concept version of the dashboard

This first layout was able to achieve the following:

  • Retrieve data from GitHub
  • Format the data to common JSON
  • Display the build statuses
  • Display the creator & assignee

While the functionality was the bare minimal it was able to achieve what we had set up in the first step of the process.

What I've also explained in the presentation I gave at the end of my internship some things took me a short while to get used to. My first pull request for example was StyleCI, which fixed some styling errors I had in my code.

We use conventions so that all the code within the organization looks the same. This way anyone can work on any code. All the code is formatted in the same fashion and is documented through docblocks. At Exonet we follow the PSR-2 standard.


Step 2

We can now retrieve pull requests from GitHub and tasks from Jira, this data is displayed on a limited page which is not styled according to the styleguide of the organization. In step 2 we're going to style the dashboard, update the frontend to use React and implement the live-updates.

In the beginning of March a designer came by to ask about our wishes for the dashboard and designed the dashboard as it looks today.

After we received the design we began implementing this in our dashboard. Beforehand we we're using plain Bootstrap CSS, when switching to the new design we chose SASS as our CSS extension to ease the process of developing the new layout. Beforehand I've never used SASS so this also was a new experience for me.

What I liked with SASS is that you (kind off) don't need to constantly repeat yourself. Take for example a corporate website, they'll have a colour they wish to use for their brand. Instead of writing #3498db over and over again you could simply do the following.

// colors.scss
$brand-color: #3498db;

// homepage.scss
.link { 
    color: $brand-color;
}

// other_thing_to_be_styled.scss
.thing {
    color: $brand-color;
}

As an added bonus, when you change your brand color, you'll only have to do that once.

As mentioned before the goal was also to change the front-end to React in the second step. Before changing the frontend of the dashboard in its entirety we first thought out a structure for the components within React. As you might already know, React is component based. This is one of the structural designs we first thought out.

<Card>  
  <Header /> 
    <Body> 
      <Info> 
      <Interaction> 
    </Body> 
</Card>  

As you might have noticed the parameters of the cards are much alike, Jira and GitHub both have a title, a creator, a reference, etc. When re-building the frontend in React we took this in mind and designed the components so that they could be used by both Jira and GitHub. This will also give us the flexibility to add more cards later which use the components that we are creating now.

import React, { PropTypes } from 'react';

/**
* Renders the user image with a given source and type. In this case
* type refers to either 'assignee' or 'creator'.
*
* @return {XML} Returns the rendered user image.
*/
const UserImage = (props) => (  
  <img
    src={props.source}
    className={`user ${props.type} img-circle`}
    alt="A display of the user"
  />
);

/**
 * @type {Object} The propTypes required to render this component.
 */
UserImage.propTypes = {  
  source: PropTypes.string.isRequired,
  type: PropTypes.string.isRequired,
};

export default UserImage;  

Above you'll find the UserImage component that we use to display user images, this is by far the least complicated component but is a perfect example of a reusable component. This is the actual component that we use and renders the assignee & creator for GitHub and the creator for Jira cards.

This step also includes the live updates over our Socket, again, entirely new. We set up a Socket server in NodeJS and after overcoming some initial bugs the live updates were (partially) working. The reason I'm saying partially is because at first we'd listen to every single event which we received over the socket and reload the entire page. When for example a build status changed to pending we'd just refresh all the pull requests so the latest change would be reflected.

Because the initial page load shows so much detail about all information on the page it's also quite rate-limit unfriendly. Because we do not receive all the data we need with a single call, we do quite a lot of them. So refreshing the dashboard on any change was not suitable. Before I even started with the dashboard this was already a known fact so the plan from the beginning on was to parse the incoming webhooks from GitHub and display the retrieved data immediately instead of refreshing all the GitHub data.

Whilst the approach was not suitable for the final dashboard, for now it was a fitting solution. This is because it made the dashboard into a usable product. Since the end of step 1 we've been using the dashboard to check if there were pull requests (first the dashboard refreshed all the data every 10 seconds or so). By adding this semi-live updates it made the dashboard into an even more useful product since the data was updated on the moment that something happened. (And, it was very cool to post a comment and see it reflected on the dashboard seconds later).

At first I wrote the parser of the events as a gigantic if if if statement. If the event is a title change, update the title. If the event is a build status update, update the build status, etc. Whilst it's quite clear what you're doing here it's not very flexible since adding a new event takes a lot of time, and the ifs were located on multiple places which made adding events even harder.

// Fire PullRequestUnassigned if user is unassigned from a pull request.
        if ($helper->isPullRequest('unassigned')) {
            $event = 'UnassignedEvent';
            $eventName = 'unassigned';
            // etcetera
        }

A small piece of the initial if-if-if-giant.

Last and final step

After I had discussed this with my colleagues, one of them decided to create some example code on how to handle these events in a more flexible way. The problem I'd created for myself was that the application was not as flexible anymore and adding events required several steps. This is a perfect example of the need to zoom-out sometimes, oversee the entire product/development and find optimizations.

A goal we had also set for ourselves was that the application should be very flexible and transferrable to colleagues. They should be able to add events in a jiffy.

A DataObject is an object in which we define parameters for a given entity. Take for example an animal, a cat and a dog both have legs, paws, ears and a tail. Yet they look different. If we were to make a dataobject for these animals they would have the ->getLegs(), ->getPaws(), ->getEars() and ->getTail() methods. The dataobject is the same yet their values differ.

To learn more about dataobjects check out this post.

As I said earlier a coworker created some example code, what this did was splitting the process of handling the events into seperate 'components' and re-use them throughout the application. I know this sounds somewhat abstract, but I'll try to explain this a little more.

Before, the application had several places where GitHub data was handled. On initial load the application used a specific function to retrieve and format the data and return this to the client. When a webhookhandler received a request, it would do the same thing yet on a different place. If GitHub changed their data format or if we wanted to change the way data was returned we would have needed to change the code on several places.

As you can imagine this is not optimal. The solution my coworker came up with in his example code was the use of DataObjects.

Underneath is the flow when initial data is retrieved. The HTTP-request comes in, for example to /pulls to retrieve all open pull requests. The controller will then ask the GitHub-class for all the open pull requests ($github->getPullrequests('open')), the GitHub class in turn will then query the GitHub API for the requested data and parse it into the PullRequestDataObject, this DataObject is then sent to the data formatter, which knows how the frontend wants the JSON and knows which methods are available for the dataobject. After the formatting the data it then sent back to the client so it can be displayed.

Below you'll find the flow for an incoming webhook received from GitHub. As you can tell there are some similarities.

  • Both flows use the same DataObject
  • Both flows use the same formatter

For the DataObject the advantage is that we already know which methods we can query, for example for a pull request we can always call $pullRequestDataObject->getTitle(). And because we use the same dataformatter we can assume the data is always returned in the way we want to.

As an added bonus now all the steps in the process from retrieving data to giving it back to the client is done in seperate components. If for whatever reason something in the chain changes we'll only need to change this once.

The difference between the initial load and the webhook handler is that on initial load the controller returns all the requested data and with the webhook handler their is nothing returned but only an event fired.


After we finished the steps

After we completed all the steps in the development process we had thought out, the end of my internship was in sight. We decided to implement some bug-fixes and further optimize some code.

One fix in particular had to do with GitHub announcing support for multiple assignees. The dashboard was crafted to handle only one assignee, when GitHub changed this the dashboard couldn't cope with the change. When 1 person was assigned, the dashboard handled just fine. When 2 people were assigned, only one of them would show up. If 2 people were assigned and one of them was unassigned, the dashboard hid both of the assignees, leaving us with the impression no one was assigned.

This bug occurred because we assumed an unassigned event would mean no one was assigned and just removed the entire assignee-parameter from our JSON in the frontend. To prevent this behavior we removed assignee in its entirety and replaced it with assignees.

Fun facts

  • 159 pull requests have been closed since the start
  • >700 commits have been pushed to the dashboard repository
  • ∞ cups of coffee have been drunk
  • Over the course of the project I've added or changed >12 000 lines of code
  • The verdict for my entire internship was: excellent, which is the highest score possible

Ruben Gommers

Read more posts by this author.

Doetinchem, Gelderland https://rubeng.nl

Subscribe to ruben gommers blog

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!