Guided: Concurrent Programming in Go

Scenario

Imagine you are part of a global hypermarket store chain operating multiple stores across different regions. Each day, you receive a summary of the sales achieved by every store, categorized by region.

However, to get a holistic view of the company's performance, you need to calculate the total sales across all stores. With a growing number of stores and increasing sales data, the current sequential approach to calculating total sales is becoming time-consuming and inefficient.

Your task is to develop a concurrent program in Go that optimizes the process of calculating total sales. By leveraging Go's powerful concurrency features, your program should efficiently compute the total sales across all stores, providing faster insights into the company's performance.

---

Code Lab

In this lab, you will dive into the world of concurrent programming in Go, exploring the power of goroutines and channels to create efficient and concurrent applications.

Beginning with fundamental Go concepts, you'll steadily advance towards integrating concurrent features. By the end, you'll have built a robust concurrent application, leveraging the full potential of Go's concurrency mechanisms.

Completing this Go lab will involve mastering the following key concepts:

Goroutines:
Learn to create lightweight concurrent threads (goroutines) for concurrent execution.

Channels:
Understand the concept of channels for communication and synchronization between goroutines.

Error Handling in Concurrent Programs:
Learn effective error handling strategies in concurrent programs.

Control Flow:
Utilize the control flow mechanism such as select statement for non-blocking communication with channels.

By the end of this lab, you will have a deep understanding of concurrent programming in Go and will have built a concurrent application, showcasing your mastery of Go programming concepts and concurrent application development. Before you begin, here are some key points:

1. Your task involves implementing code in the main.go file.
2. If you encounter any challenges along the way, feel free to consult the solution directory.
3. To simplify the process, comments are included to help you find the necessary changes for each task.
4. You can execute the program using the Run button or by running the go run . command in the Terminal.

Concurrency in Go is a fundamental aspect of the language's design, empowering developers to write efficient, scalable, and responsive software. Go's concurrency model revolves around goroutines and channels, providing a powerful yet simple way to handle concurrent tasks.

Before you get started, it's a good idea to learn about the files that you'll be working with and the default code in those files.

The salesdata.go File

This file contains the sample sales data grouped by each stores. If you open the salesdata.go file, each row signifies a region and every value is the sales of the particular store in that region.

### The `main.go` File This file contains the code which calculates the total sales across different regions.

Inside the main function, there's a loop over salesData, which contains sales data for different regions. For each region, it calls the calculateRegionSales function, passing the sales data for that region and a pointer to the totalSales.

The calculateRegionSales function calculates the total sales for a given region. It iterates over the sales data for the region, adding each store's sales to the totalSales.

To better understand the impact of concurrency, there is a delay of 100 milliseconds added for each store's sales calculation using time.Sleep.

The main method also prints the total time taken to calculate the sum once the calculation is complete. This will help you compare the execution time before and after the concurrency changes.

---

This code computes the sum incrementally, processing one region at a time.

You can try executing this code by clicking on the Run button or by typing the go run . command in the Terminal tab.

Output:

Total 50000000, Time taken to calculate 5.xxxs

To write concurrent programs, it's important to understand goroutines.

Goroutines

Goroutines are lightweight threads managed by the Go runtime. They enable concurrent execution of functions, allowing multiple tasks to run simultaneously within a single Go program. Goroutines are incredibly cheap in terms of memory footprint and overhead, making it practical to spawn thousands of them within a single application.

To create a goroutine, you simply prefix a function call with the go keyword. For example:

func main() {
   // Start a new goroutine
   go sayHello()

   // Continue with main execution
   fmt.Println("Main function")

   // Sleep for a while to allow the goroutine to finish
   time.Sleep(1 * time.Second)
}

func sayHello() {
   fmt.Println("Hello from goroutine!")
}

Output:

Main function
Hello from goroutine!

In the above example, the sayHello() function is executed concurrently as a goroutine.

Because goroutines operate concurrently in distinct paths, it's essential to acknowledge that if the main thread ends, all goroutines also terminate. To prevent premature termination of the main thread, a one-second delay is incorporated using time.Sleep. This pause enables the goroutine to finish its execution smoothly.

---

Click on the Run button to execute the program.

Well done! You've effectively employed a Go routine to accomplish the task in nearly 3 seconds, a significant improvement over the sequential 5 seconds.

Even though this works, there are couple of shortcomings with this approach:

1. When the CPU is busy, your program's execution might exceed the 3-second threshold, possibly causing goroutines to terminate prematurely within that 3 seconds time frame. This unpredictability could lead to inaccurate totals.

2. Even if your program finishes in less than 3 seconds, you still have to wait for 3 seconds for the output to generate.

Ideally, your program should terminate as soon as all the goroutines are finished.

In the next step, you will address all of the aforementioned issues and optimize the code.

In the previous step, you examined the use of goroutines for executing tasks concurrently. Nonetheless, the challenge persisted in ensuring that the main goroutine concludes its execution promptly immediately after the goroutines finish executing.

In this step, you will see how WaitGroup solves this problem.

Wait Group

Wait groups are a synchronization mechanism provided by the sync package in Go. They allow you to wait for a collection of goroutines to finish their execution before proceeding further in the program. Wait groups are particularly useful when you have a dynamic number of goroutines and need to ensure they all complete their tasks before continuing.

How Wait Groups Work

Wait groups are represented by the sync.WaitGroup type. You create a new wait group using var wg sync.WaitGroup, and then add the number of goroutines you want to wait for using the Add() method.

Each goroutine increments the wait group counter using wg.Add(1) before starting its task. When a goroutine completes its task, it decrements the counter using wg.Done().

In the below example, two goroutines are added to the wait group using wg.Add(2) before their execution begins.

Finally, the main goroutine, or any other goroutine waiting for the completion of the tasks, calls wg.Wait() to block until all goroutines have finished their tasks.

func main() {
   var wg sync.WaitGroup

   // Add two goroutine to the wait group
   wg.Add(2)

   // Start goroutines
   go doSomeWork(&wg)
	 go doSomeWork(&wg)

   // Wait for all goroutines to finish
   wg.Wait()
   fmt.Println("All goroutines have finished")
}

func doSomeWork(wg *sync.WaitGroup) {
   // Signal that the goroutine has finished
   defer wg.Done() 
   // Simulate some work
   fmt.Println("Goroutine: Working...")
   time.Sleep(time.Second)

   fmt.Println("Goroutine: Finished!")
}

In this example, defer wg.Done() is used inside the doSomeWork function. The defer ensures that no matter how the function exits, whether it returns normally or panics, it will always call wg.Done() before exiting. This helps ensure that the WaitGroup counter is decremented appropriately, allowing wg.Wait() in the main function to block until all workers are finished.

Since doSomeWork is invoked twice, it will decrement the counter twice using wg.Done() . Once the WaitGroup counter is decremented to 0, it will unblock the wg.Wait() call.

This pattern ensures that you properly wait for all goroutines to finish before proceeding, even if an error occurs within a goroutine.

---

Now, you will optimize the code using wait groups to ensure your program completes faster. Click on the Run button to execute the program.

Kudos! You are successfully able to now calculate the totalSales in around 1 second!

It's possible for two goroutines to simultaneously update the totalSales variable, leading to inconsistent totals. To avoid this, Go offers synchronization primitive called mutex, which is used to control access to shared resources. It ensures that only one goroutine can access the shared resource at any given time, thus ensuring consistency.

You'll explore more about it in the next step.

In this step, you'll learn about mutex. You'll understand and implement mutex to ensure that the totalSales variable is updated only by one goroutine at a time.

Mutex

A mutex, short for mutual exclusion, is a synchronization primitive used to control access to shared resources in concurrent programs. It ensures that only one goroutine can access a shared resource at a time, preventing data races and ensuring consistency.

In Go, a mutex is represented by the sync.Mutex type from the sync package. It provides two main methods:

• Lock() : Acquires the mutex. If the mutex is already locked by another goroutine, Lock() will block until it becomes available.
• Unlock() : Releases the mutex, allowing other goroutines to acquire it.

To use a mutex in your Go code, follow these steps:

• Declare a variable of type sync.Mutex.
• Use Lock() to acquire the mutex before accessing the shared resource, and Unlock() to release it afterwards.

Example :

var (
    counter int
    mutex   sync.Mutex
)

func increment() {
    mutex.Lock()    // Acquire the mutex before modifying the counter
    counter++       // Increment the counter
    mutex.Unlock()  // Unlock the mutex after modifying the counter
}

func main() {
    var wg sync.WaitGroup

    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            increment()
        }()
    }

    wg.Wait()

    fmt.Println("Counter:", counter)
}

In this example:

• It has a shared variable counter and a mutex mutex.
• The increment() function locks the mutex before incrementing the counter and unlocks it afterwards.
• In the main() function, it spawns 10 goroutines, each calling increment() concurrently.
• It waits for all goroutines to finish using sync.WaitGroup.
• Finally, it prints the value of counter.

---

Now, you will optimize the existing program to synchronize access to totalSales using mutex. Great! Now that you've made the sales calculation synchronized, it will ensure that you get accurate total sales number every time.

Up to this point, you've been utilizing wait groups to facilitate concurrent processing of calculations. However, another powerful mechanism in Go for managing concurrency is channels.

There are certain scenarios where channels might be preferred over wait groups.

Here are some common issues with wait groups that might lead to the use of channels instead:

Complexity:

• Managing concurrency with wait groups can lead to intricate code, especially when dealing with nested or dynamic concurrency patterns.

Error Handling

• Wait groups lacks built-in error handling mechanisms, necessitating manual propagation of errors through other means, potentially leading to code complexity and increased error risk.

Unordered Completion

• Wait groups doesn't ensure the ordered completion of goroutines. Processing results in a specific order requires additional synchronization mechanisms, adding complexity to the code.

Difficulty in Dynamic Scaling

• Dynamically managing the number of goroutines with wait groups poses challenges, as careful addition and subtraction from the wait groups are required, risking race conditions if not executed correctly.

In the next step, you'll look how channels solve the aforementioned problems.

In this step, you are going to learn about channels. Compared to wait groups, channels offer a more flexible and expressive way to manage concurrency:

Synchronization

• Channels offer built-in synchronization, ensuring safe data transmission between goroutines without extra synchronization primitives.

Error Handling

• Channels facilitate error propagation alongside data, simplifying error management and reducing error susceptibility.

Ordering

• Channels enforce operation sequence, guaranteeing correct result processing without added synchronization overhead.

Dynamic Scaling

• Channels excel in scenarios with unknown or dynamic goroutine counts. They enable on-demand goroutine creation and coordination via channels.

Now that you understand the benefits of channels, you'll delve into the basics of how channels work.

Channels serve as the primary means of communication and synchronization between goroutines in Go, providing a safe conduit for sending and receiving data. They enable concurrent processes to exchange values without the need for explicit locking or coordination, ensuring that operations proceed smoothly without interference or race conditions.

Creating Channels

Channels are created using the make function with the chan keyword followed by the type of data that the channel will transmit. Here's how you create a channel:

// Creates an unbuffered channel of type int
ch := make(chan int) 

Channel Operations

Channels support two main operations: sending and receiving values. These operations are performed using the <- operator:

ch <- value // Send value into the channel
value := <-ch // Receive value from the channel

Buffered Channels

By default, channels are unbuffered, meaning they only accept a value if there's a corresponding receiver ready to receive it. Buffered channels, on the other hand, have a fixed capacity and can store a certain number of values without a corresponding receiver. Here's how you create a buffered channel:

ch := make(chan int, bufferSize) // Creates a buffered channel with capacity bufferSize

Closing Channels

Channels can be closed to indicate that no more values will be sent. Receivers can use the second return value from a receive operation to determine if the channel has been closed:

close(ch) // Close the channel

Channel Direction

You can specify the direction of a channel in its type signature to restrict its usage to sending or receiving operations. This helps enforce communication protocols and prevent misuse of channels. Here's how you specify channel direction:

func sendData(ch chan<- int) {
  // Send data into channel
}

func receiveData(ch <-chan int) {
  // Receive data from channel
}

---

Here's a simple example demonstrating how to use channels for communication between two goroutines. In this case, the sendData function sends integers to the channel, and the main function receives and prints those integers.

func sendData(ch chan<- int) {
 // Send data into the channel
 ch <- 10
 ch <- 20
 ch <- 30
 close(ch) // Close the channel after sending all values
}

func main() {
 // Create an unbuffered channel of type int
 ch := make(chan int)

 // Start a goroutine to send data into the channel
 go sendData(ch)

 // Receive data from the channel
 for {
   // Attempt to receive a value from the channel
   value, ok := <-ch
   if !ok {
   // Channel closed, exit the loop
   break
  }
  // Print the received value
  fmt.Println("Received:", value)
  }
}

Output:

Received: 10
Received: 20
Received: 30

---

Now that you understand channels, you'll also understand the difference in the approach of calculation.

In the previous approach with wait groups and goroutines, you were using a common variable totalSales, which was being incremented by each goroutine concurrently.

With channels you will let each goroutine calculate the total of the region and send it back to you in a channel, salesCh. Once the regionTotal is received from channel, you'll add them to the totalSales variable.

Now, you'll update the code to use channels instead of a wait group. Click on the Run button to execute the program.

In the next step, you'll learn about handling errors in concurrent programs using channels.

In Go, channels can be used not only for communication between goroutines, but also for propagating errors. By sending error values through channels, goroutines can report errors to other parts of the program or to the main goroutine responsible for error handling:

 // Create a channel for error communication
 errCh := make(chan error)

Example of error propagation using channels:

func doTask(resultCh chan int, errCh chan error) {
 resultCh <- 42              // Simulate a calculation
 errCh <- errors.New("something went wrong") // Simulate an error
}

func main() {
 resultCh := make(chan int)  // Channel for result
 errCh := make(chan error)   // Channel for error

go doTask(resultCh, errCh)  // Start the task goroutine

 select {
  case result := <-resultCh:
   fmt.Println("Result:", result)
  case err := <-errCh:
   fmt.Println("Error:", err)
  }
}

In this example:

• The doTask function sends the result, 42, and an error, ("something went wrong"), directly to their respective channels.
• Channels for both the result and the error communication are created.
• The main function starts the task goroutine using go doTask(resultCh, errCh).
• The select statement handles either receiving the result or the error from their respective channels and prints them accordingly.

---

You've observed that sometimes data for regions come incorrect and sales number are received with negative values. In those cases, the data for the region should be discarded and the error should also be reported.

Now, you'll update the program to include the error scenario of negative sales. Click on the Run button to execute the program.

Excellent work! You've grasped the fundamental concepts of concurrency in Go.

You've increased the calculation speed five-fold. Now, picture a scenario with much more data, involving 5000 stores across numerous additional regions. In this scenario, rather than requiring five hours, the processing would be finished within just one hour. This signifies the vast potential of concurrency.

Congratulations on completing the Concurrent Programming in Go lab! Throughout this journey, you've explored the fascinating world of concurrent programming in Go, leveraging powerful features like goroutines and channels to build efficient and scalable applications.

You began by understanding fundamental Go concepts, gaining proficiency in syntax and language features. From there, you delved into the heart of concurrency, learning to create lightweight threads (goroutines) for concurrent execution. You then explored channels, understanding their role in facilitating communication and synchronization between goroutines.

Error handling in concurrent programs was another key aspect you tackled, learning effective strategies to handle errors gracefully in a concurrent context. Finally, you utilized control flow mechanisms like the select statement to enable non-blocking communication with channels, further enhancing the efficiency of your concurrent applications.

Next Steps:

Now that you've completed the lab, here are some suggested next steps to further solidify your understanding and expertise in concurrent programming in Go:

1. Explore Advanced Topics

• Dive deeper into advanced concurrency topics such as synchronization primitives, race conditions, atomic operations, and context handling in Go.

2. Learn about Concurrency Patterns in Go

• Go provides several concurrency patterns to help developers write efficient and scalable concurrent programs.

As you continue your Go learning journey, consider exploring the Go path on Pluralsight:

Pluralsight Go Path

Keep exploring, experimenting, and honing your Go skills. With dedication and practice, you'll continue to grow as a proficient Go developer. Best wishes on your coding endeavors!