initialCapacity[]

Fundamentals of Software Architecture for Big Data

The course is intended for individuals looking to understand the basics of software engineering as they relate to building large software systems that leverage big data. You will be introduced to software engineering concepts necessary to build and scale large, data intensive, distributed systems. Starting with software engineering best practices and loosely coupled, highly cohesive data microservices, the course takes you through the evolution of a distributed system over time.

← Back

Download the codebase

Email Verifier

An app for verifying email addresses in a registration flow, which is designed to handle very high throughput.

Set up

  1. Run docker-compose.

    docker-compose up
    
  2. Run migrations

    ./gradlew devMigrate testMigrate
    

Build and run

  1. Use the Gradle Kotlin plugin to run tests, build, and fetch dependencies. For example, to build run

    ./gradlew build
    
  2. Run the notification server.

    ./gradlew applications:notification-server:run
    

    Luckily, Gradle fuzzy-matches task names, so the command can optionally be shortened to

    ./gradlew a:n:r
    
  3. Run the registration server in a separate terminal window.

    ./gradlew applications:registration-server:run
    
  4. Run the fake Sendgrid server in another separate terminal window.

    ./gradlew platform-support:fake-sendgrid:run
    

Make requests

  1. Post to http://localhost:8081/request-registration to make a registration request. Include the email address to register in the request body.

    {
      "email": "[email protected]"
    }
    

    Don't forget to add the content type header.

    Content-Type: application/json
    
  2. Check the logs of the fake Sendgrid server for your confirmation code. Once you receive it, post to http://localhost:8081/register to confirm your registration. Include your email address and confirmation code in the request body.

    {
        "email": "[email protected]",
        "confirmationCode": "18675309-1234-5678-90ab-cdef00000000"
    }
    

    Don't forget to add the content type header.

    Content-Type: application/json
    

See the requests.http file for sample requests

Benchmarks

The benchmark app runs a simple benchmark test against the running apps.

  1. Stop the fake Sendgrid app, then run the benchmark app with

    ./gradlew applications:benchmark:run
    

    This will send some traffic to the notification and registration servers, and will print some basic metrics to the console.

  2. Once the benchmark is finished, try running it again giving different values for the REGISTRATION_COUNT, REGISTRATION_WORKER_COUNT, and REQUEST_WORKER_COUNT environment variables.

  3. After getting comfortable with the environment, try running multiple instances of the notification server and the registration server. Make sure to provide a unique PORT environment variable to each instance of the registration server.

Consistent hash exchange

Now that we have our system working with multiple instances, we will implement a consistent hash exchange to better distribute load between our registration request consumers. Look for the TODOs in the codebase to help you get started.

A workshop by

Initial Capacity