Before we started working on the web portal at work, we discussed which languages to use. We had a team of four developers and planned to hire more. We considered the hiring pool, package availability, language standardization, testing, and scalability.

PHP

A coworker suggested PHP and Laravel. We would have access to a nice pool of web developers so there would be no shortage of talent. Laravel is also well documented. I spent my first couple weeks at the company learning Laravel for a different project and I wasn’t building as fast as I had hoped, even with my six years of experience in PHP. Autocomplete was difficult to set up so I spent a lot of time in the documentation looking up commands. There was also a lot of magic in the background that made it difficult to troubleshoot.

Node

Also on the table was Node. A lot of interactive applications use Node and we could create dynamic applications that looked great and performed well, but no one on the team had built anything other than a “hello world” application. JavaScript was also not any of our top languages. Even though we didn’t choose Node, npm (Node Package Manager) turned out to be an asset for tasks like compiling SASS and minifying JavaScript.

Go

Finally, we discussed Go. The two big issues were the small hiring pool and the availability of packages.

To address the first one, Go is not a difficult language to learn and you can be productive quickly. You can read the entire spec very quickly and you can walk through the tutorials in a few hours. The low number of keywords means it’s easier to start working with the language and you’re more likely to pick the “best” way to complete a task. Since the source of the language is mostly written in Go, you can use that code as a resource.

The second issue is not an issue, it’s a mindset. We all want rapid application development (RAD), but how beneficial is a “RAD only” language in the long run? Go believes in small packages, not frameworks. A small number of dependencies means a smaller number of bugs and more stability (generally). RAD is great for prototyping, but it’s the 20% of the features that takes up 80% of your time. Many languages and frameworks give you the 80% of the features you need for free and the other 20% you spend your time figuring out or writing yourself. Go has a few big web frameworks, but there is no reason to use them because you can find smaller packages to fit your needs. We also wanted our applications to be modular so if we used a web framework for one application, it would be difficult to use the same packages in another daemon in another application.

We found most of what we needed in the Go standard library. It includes packages which allow you to build on top of a web listener, parse form variables, compile templates - all required when building a web site. As long as we had access to the AWS SDK and a good LDAP package, we had the minimum requirements for the web portal.

The team decided to use Go because we could use it for the web portal and other applications, it was quick and scalable, and not likely to change or go obsolete in the next 2-3 years because of the version 1 promise.

Our Experiences

Standardizing on a Language

Our team was mostly DevOps Engineers - self included. The web portal was not our only application. We had to interact with other systems, logs, etc. Many of the applications were daemons that ran every 5 or 10 minutes to sync users and groups from Active Directory to another application or load log files into another application for analysis. This is where standardizing was extremely valuable. We wrote all of daemons in Go. Why? Anyone on the team could build and work on them. No to mention that Go runs on Windows, Linux, and OS X so we didn’t have to worry about multiple platforms slowing our work. This also meant we could reuse our code between applications. For small applications, we copied our configs and models into the new repository and were ready to go. Every application used a configuration file that followed the same format so that it was easy to reuse and make changes to environment variables.

Troubleshooting

Our team only hit a few issues where we spent over an hour debugging and it was because a few of our database queries were not written to handle one-to-many relationships and we didn’t handle the “no rows” error correctly from the model. We learned our lesson and started handling the “no rows” error separate from the other errors. In Go, you have control over almost every error (race conditions are an exception) so you don’t have to wrap your code in try/catch blocks because you know which functions could potentially fail. Hitting on the race condition piece above, you can write test cases that check for race conditions so it’s not an issue. In an enterprise application, you should not be guessing what will happen. You should be able to build a function that returns the value you want or returns an error that you handle appropriately.

Testing

Since we used the same language for all our applications, they shared the same continuous integration and continuous deployment (CI/CD) strategy. Tests are important to write because they make you an effective programmer by forcing you to write testable code. Testable code is stable code. We used the GitLab Runners to automatically run the tests when we committed code.

As applications get larger, you need to be able to test pieces individually. We built a lot of automation into our code like user creation. When a user was created, we had to perform about five other steps. When we were testing, it made a lot more sense to write test cases for each of those features. You shouldn’t have to run through the entire process of five steps just to test one stage of them.

Scalability

Scalability is not an issue with Go. We don’t have to worry about a thread blocking and hanging up the entire web application because the user was trying to upload a large file. Go handles every request on a new thread so the application continues to run smoothly. We built an application where users can upload large files and then those files are analyzed. The application was a rewrite from a Python application that failed with memory errors. Once we rewrote it in Go (took 2 days), we didn’t have any more issues. I’m sure a solid Python developer could have fixed the memory issues, but features like multi-threading are important in web applications so Python wasn’t best suited for the task.

Dependencies

When working by yourself, go get is perfectly fine. The difficulty comes when you commit code into a repository that is worked on by many developers and you have no idea if those external packages are going to change when you deploy to production. The vendor folder is great, but you must copy your packages into that folder after a go get. We always had more than one project in our Go workspace at a time so the go get would mix the new packages with existing packages We tried using govendor, but developers on our team had problems restoring dependencies when syncing their branches with master so we switched to gvt which worked well for us. Run gvt fetch [repo name] and the packages will download to the vendor folder. Then, commit the vendor folder into the repository with the code. This ensures every application compiles reliably from the master branch. To update the dependencies, just run gvt update.

Branching

We used a typical workflow for making changes to the code. If you worked on a feature or a bug, create a branch from master, make your changes, and then submit a merge request. The team lead would review the merge request, and then merge into the master branch. Then we let the team know when we changed the config file or added a new migration so team members know when to update their local dev environment. This is not specific to Go, but the workflow worked well for us with Go.

Database Migrations

Again, not specific to Go, but just our results dealing with the migrations. We stored database migrations in the repo with the application source code. One problem we found was the migrations were out of order if two people worked on their own branch and both created a new database migration. If we merged the newest database migration first, the other developer would have to change the date on their database migration so it was ordered AFTER all the other migrations. Otherwise, the production server would skip over that migration. This was a limitation of the migration tool we used: Jay. The issue didn’t happen often, but it did require some coordination by the developers.

Effective Logging

There are two ways developers can output information: fmt.Println() and log.Println(). Here is what they look like in the console:

fmt.Println("Hello world") // Output: Hello world
log.Println("Hello world") // Output: 2009/11/10 23:00:00 Hello world

The fmt package is not helpful with error messages because it does not tell you when or where the message is being called from. The log package does tell you the time the message was printed, but it doesn’t tell you where.

We used the log.SetFlags(log.Lshortfile) command in all of our applications to see where the message was printed from when using log:

package main

import (
	"log"
)

func init() {
	// Verbose logging with file name and line number
	log.SetFlags(log.Lshortfile)
}

func main() {
	log.Println("Hello world") // Output: main.go:13: Hello world
}

We also added this line to our init() function so error messages would not display on the Linux console and could be logged easily:

// Log error to standard out
log.SetOutput(os.Stdout)

This is the command we used to log the application output to a file:

./main >> output.txt &

Error Handling

When we started developing in Go, one thing we noticed was a lot of the developers were not handling errors properly. Specifically, the errors from the models. All of our models returned a struct with database results and an error.

Here is an example:

// Error should be returned in the second position, but it is ignored
data, _ := user.ByID("4")

The developers didn’t know what to do with the errors so they suppressed them. It took some training on error handling to correct this. Some errors were in the query syntax while others were returning no rows and after a few of these queries in a row, it took longer to figure out where the issue was.

Overall

The company is really happy with Go. The developers enjoy the language and we haven’t had any problems we couldn’t solve. Management is impressed with how quickly we deliver and one of the founders is learning Go so he can contribute. We now have around eight applications that use Go in production.

To address the initial issues, we found packages when we needed them. We interfaced with five different APIs - some using SDKs and other interacting directly via REST (JSON) or SOAP (XML). None of our developers except me had any Go experience, but they all picked it up quickly so the potential issue of a small hiring pool is not actually an issue. All we needed was good developers ready to learn a new language.

We used Blue Jay as a starting point for many of our applications. We didn’t have to spend time setting up a website framework and could follow good practices from the start.