Author avatar

Stefan Prodan

Build a scalable, fault tolerant system with ASP.NET Core and RethinkDB on Docker Swarm mode

Stefan Prodan

  • Jan 10, 2019
  • 25 Min read
  • 10,143 Views
  • Jan 10, 2019
  • 25 Min read
  • 10,143 Views
Microsoft.NET

Introduction

This guide shows you how to use RethinkDB with ASP.NET Core and how to deploy your app along with a RethinkDB cluster using Docker for Windows and Docker Swarm mode. We will be using the TokenGen app from the Scale ASP.NET Core apps with Docker Swarm Mode guide and store the generated tokens along with the issuers in a RethinkDB database.

RethinkDB is an open source distributed document-oriented database. As most NoSQL databases, it stores JSON documents with no required schema or table structure.

What makes RethinkDB great:

  • horizontal scaling
  • table replication and sharing
  • automatic fail-over
  • query language with join support, map-reduce and geospatial queries
  • pub/sub for data changes

If you want a scalable, fault tolerant database system for your ASP.NET Core applications then you should consider using RethinkDB.

Setting up a RethinkDB Cluster with Docker Swarm Mode

First we need to enable Swarm Mode and create a dedicated network for our RethinkDB cluster:

1# initialize swarm
2docker swarm init
3
4# create RethinkDB overlay network
5docker network create --driver overlay rdb-net
powershell

We start building our RethinkDB cluster by running a single RethinkDB server, we will remove this instance later on:

1# create and start rethinkdb primary
2docker service create --name rdb-primary --network rdb-net --replicas 1 rethinkdb:latest rethinkdb --bind all --no-http-admin
powershell

Now we can create a secondary RethinkDB node that will join the rdb-primary node and form a cluster:

1# create and start rethinkdb secondary
2docker service create --name rdb-secondary --network rdb-net --replicas 1 rethinkdb:latest rethinkdb --bind all --no-http-admin --join rdb-primary
powershell

Scale the secondary node so we can have a minimum of 3 nodes needed for RethinkDB automatic fail-over mechanism:

1# up 3 nodes (primary + two secondary) to enable automatic failover
2docker service scale rdb-secondary=2
powershell

We now have a functional RethinkDB cluster, but we are not done yet. Because we started the primary node without a join command, our cluster has a single point of failure. If for some reason rdb-primary container crashes, the Docker Swarm engine will recreate and start this container, but he can't join the existing cluster. If we start new rdb-secondary instances, they will join the new rdb-primary container and form another cluster.

To resolve this issue we have to remove the rdb-primary service and recreate it with the join command like so:

1# remove primary
2docker service rm rdb-primary
3
4# recreate primary with --join flag
5docker service create --name rdb-primary --network rdb-net --replicas 1 rethinkdb:latest rethinkdb --bind all --no-http-admin --join rdb-secondary
powershell

Now we can also scale the primary node:

1# start two rdb-primary instances
2docker service scale rdb-primary=2
powershell

At this point we have 4 nodes in our cluster, two rdb-primary and two rdb-secondary. We can further scale any of these two services and they will all join our cluster. If a rdb-primary or rdb-secondary instance crashes, the Docker Swarm will automatically start another container that will join our current cluster.

Last step is to create a RethinkDB proxy node, we expose port 8080 for the web admin and port 28015 so we can connect to the cluster from our app. Note that you don't have to expose port 28015 if your application runs inside the swarm. I'm exposing this port so I can connect to the cluster while debugging in VS.NET.

1# create and start rethinkdb proxy
2docker service create --name rdb-proxy --network rdb-net --publish 8080:8080 --publish 28015:28015 rethinkdb:latest rethinkdb proxy --bind all --join rdb-primary
powershell

Open a browser and navigate to http://localhost:8080 to check the cluster state. In the servers page you should see 4 servers connected to the cluster.

if we run docker service ls we should get:

1ID            NAME           REPLICAS  IMAGE             COMMAND
2157bd7yg7d60  rdb-secondary  2/2       rethinkdb:latest  rethinkdb --bind all --no-http-admin --join rdb-primary
341eloiad4jgp  rdb-primary    2/2       rethinkdb:latest  rethinkdb --bind all --no-http-admin --join rdb-secondary
467oci5m1wksi  rdb-proxy      1/1       rethinkdb:latest  rethinkdb proxy --bind all --join rdb-primary

Connecting to RethinkDB from ASP.NET Core

Open the TokenGen project from the previous guide and install the RethinkDB driver NuGet package:

1Install-Package RethinkDb.Driver

We are going to create an object that holds the RethinkDB cluster address. Add a class named RethinkDbOptions inside TokenGen project:

1public class RethinkDbOptions
2{
3    public string Host { get; set; }
4    public int Port { get; set; }
5    public string Database { get; set; }
6    public int Timeout { get; set; }
7}
csharp

We will be storing the connection data in appconfig.json and use a RethinkDbOptions object to inject the data into RethinkDbConnectionFactory. The RethinkDbConnectionFactory will provide a persistent connection to the RethinkDB cluster for our app:

1public class RethinkDbConnectionFactory : IRethinkDbConnectionFactory
2{
3    private static RethinkDB R = RethinkDB.R;
4    private Connection conn;
5    private RethinkDbOptions _options;
6
7    public RethinkDbConnectionFactory(IOptions<RethinkDbOptions> options)
8    {
9        _options = options.Value;
10    }
11
12    public Connection CreateConnection()
13    {
14        if (conn == null)
15        {
16            conn = R.Connection()
17                .Hostname(_options.Host)
18                .Port(_options.Port)
19                .Timeout(_options.Timeout)
20                .Connect();
21        }
22
23        if(!conn.Open)
24        {
25            conn.Reconnect();
26        }
27
28        return conn;
29    }
30
31    public void CloseConnection()
32    {
33        if (conn != null && conn.Open)
34        {
35            conn.Close(false);
36        }
37    }
38
39    public RethinkDbOptions GetOptions()
40    {
41        return _options;
42    }
43}
csharp

Open appsettings.json and add the RethinkDB cluster connection data:

1  "RethinkDbDev": {
2    "Host": "localhost",
3    "Port": 28015,
4    "Timeout": 10,
5    "Database": "TokenStore"
6  }
json

Now we can create a singleton instance of RethinkDbConnectionFactory in Startup.cs. You should reuse the same connection across whole application since the RethinkDb connection is thread safe.

1public void ConfigureServices(IServiceCollection services)
2{
3    //....
4
5    services.Configure<RethinkDbOptions>(Configuration.GetSection("RethinkDbDev"));
6    services.AddSingleton<IRethinkDbConnectionFactory, RethinkDbConnectionFactory>();
7}
csharp

Test the connection inside Startup.Configure method:

1public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory,
2    IRethinkDbConnectionFactory connectionFactory)
3{
4    //....
5
6    var con = connectionFactory.CreateConnection();
7    con.CheckOpen();
8}
csharp

Database initialization

The TokenGen app logic involves two entities: token and issuer. Tokens are issued by an app instance (issuer) to a client by calling the /api/token endpoint. We will create a database to store these two entries.

First we create a class for each entity:

1public class Issuer
2{
3    [JsonProperty("id", NullValueHandling = NullValueHandling.Ignore)]
4    public string Id { get; set; }
5    public string Name { get; set; }
6    public string Version { get; set; }
7    public DateTime Timestamp { get; set; }
8}
9
10public class Token
11{
12    [JsonProperty("id", NullValueHandling = NullValueHandling.Ignore)]
13    public string Id { get; set; }
14    public DateTime Expires { get; set; }
15    public string Issuer { get; set; }
16}
csharp

Now we will create a service that uses the RethinkDb connection factory and implements the database initialization logic:

1public class RethinkDbStore : IRethinkDbStore
2{
3    private static IRethinkDbConnectionFactory _connectionFactory;
4    private static RethinkDB R = RethinkDB.R;
5    private string _dbName;
6
7    public RethinkDbStore(IRethinkDbConnectionFactory connectionFactory)
8    {
9        _connectionFactory = connectionFactory;
10        _dbName = connectionFactory.GetOptions().Database;
11    }
12
13    public void InitializeDatabase()
14    {
15        // database
16        CreateDb(_dbName);
17
18        // tables
19        CreateTable(_dbName, nameof(Token));
20        CreateTable(_dbName, nameof(Issuer));
21
22        // indexes
23        CreateIndex(_dbName, nameof(Token), nameof(Token.Issuer));
24        CreateIndex(_dbName, nameof(Issuer), nameof(Issuer.Name));
25
26    }
27
28    protected void CreateDb(string dbName)
29    {
30        var conn = _connectionFactory.CreateConnection();
31        var exists = R.DbList().Contains(db => db == dbName).Run(conn);
32
33        if (!exists)
34        {
35            R.DbCreate(dbName).Run(conn);
36            R.Db(dbName).Wait_().Run(conn);
37        }
38    }
39
40    protected void CreateTable(string dbName, string tableName)
41    {
42        var conn = _connectionFactory.CreateConnection();
43        var exists = R.Db(dbName).TableList().Contains(t => t == tableName).Run(conn);
44        if (!exists)
45        {
46            R.Db(dbName).TableCreate(tableName).Run(conn);
47            R.Db(dbName).Table(tableName).Wait_().Run(conn);
48        }
49    }
50
51    protected void CreateIndex(string dbName, string tableName, string indexName)
52    {
53        var conn = _connectionFactory.CreateConnection();
54        var exists =  R.Db(dbName).Table(tableName).IndexList().Contains(t => t == indexName).Run(conn);
55        if (!exists)
56        {
57            R.Db(dbName).Table(tableName).IndexCreate(indexName).Run(conn);
58            R.Db(dbName).Table(tableName).IndexWait(indexName).Run(conn);
59        }
60    }
61
62    public void Reconfigure(int shards, int replicas)
63    {
64        var conn = _connectionFactory.CreateConnection();
65        var tables = R.Db(_dbName).TableList().Run(conn);
66        foreach (string table in tables)
67        {
68            R.Db(_dbName).Table(table).Reconfigure().OptArg("shards", shards).OptArg("replicas", replicas).Run(conn);
69            R.Db(_dbName).Table(table).Wait_().Run(conn);
70        }
71    }
72}
csharp

The InitializeDatabase method checks if the TokenStore database, tables and indexes are present on the cluster, if they are missing then it will create them. Because these operations are expensive you should call this method only once at application start.

On application startup, create a singleton instance of IRethinkDbStore and invoke InitializeDatabase:

1public void ConfigureServices(IServiceCollection services)
2{
3    //...
4
5    services.Configure<RethinkDbOptions>(Configuration.GetSection("RethinkDbDev"));
6    services.AddSingleton<IRethinkDbConnectionFactory, RethinkDbConnectionFactory>();
7
8    services.AddSingleton<IRethinkDbStore, RethinkDbStore>();
9}
10
11public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory,
12    IRethinkDbStore store)
13{
14    //...
15
16    store.InitializeDatabase();
17}
csharp

When the InitializeDatabase runs, the tables are created by default with one shard and one replica. We've setup earlier a RethinkDB cluster with 4 nodes, to make our tables fault tolerant we need to replicate our data on all 4 nodes. In order to do this, after the database initialization we can call store.Reconfigure(shards = 1, replicas = 4).

CRUD Operations

When a TokenGen app instance starts it has to register in the database as an issuer, we will use the MachineName to identity the issuer. When docker starts a new container, the MachineName gets populated with the container ID, so we will use this ID to track the load of each app instance.

First we create a method to persist issuers:

1public class RethinkDbStore : IRethinkDbStore
2{
3    //..
4
5    public string InsertOrUpdateIssuer(Issuer issuer)
6    {
7        var conn = _connectionFactory.CreateConnection();
8        Cursor<Issuer> all = R.Db(_dbName).Table(nameof(Issuer))
9            .GetAll(issuer.Name)[new { index = nameof(Issuer.Name) }]
10            .Run<Issuer>(conn);
11
12        var issuers = all.ToList();
13
14        if (issuers.Count > 0)
15        {
16            // update
17            R.Db(_dbName).Table(nameof(Issuer)).Get(issuers.First().Id).Update(issuer).RunResult(conn);
18
19            return issuers.First().Id;
20        }
21        else
22        {
23            // insert
24            var result = R.Db(_dbName).Table(nameof(Issuer))
25                .Insert(issuer)
26                .RunResult(conn);
27
28            return result.GeneratedKeys.First().ToString();
29        }
30    }
31}
csharp

This method uses the Issuer.Name secondary index we created earlier to search for an issuer by name. If the issuer exists, it will update the version and timestamp, else it will insert a new one. Note that the update will only happen when you run the app from Visual Studio. On docker swarm, when you scale up a service, each instance has a unique ID.

We can now call this method on application start:

1public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory,
2    IRethinkDbStore store)
3{
4    //...
5
6    // create TokenStore database, tables and indexes if not exists
7    store.InitializeDatabase();
8
9    // register issuer
10    store.InsertOrUpdateIssuer(new Issuer
11    {
12        Name = Environment.MachineName,
13        Version = PlatformServices.Default.Application.ApplicationVersion,
14        Timestamp = DateTime.UtcNow
15    });
16}
csharp

Now that we've implemented the issuers persistence we can do the same for the tokens by adding the following method to RethinkDbStore:

1public void InserToken(Token token)
2{
3    var conn = _connectionFactory.CreateConnection();
4    var result = R.Db(_dbName).Table(nameof(Token))
5        .Insert(token)
6        .RunResult(conn);
7}
csharp

Edit the TokenController and call InsertToken before sending the token to the client:

1[Route("api/[controller]")]
2public class TokenController : Controller
3{
4    private IRethinkDbStore _store;
5
6    public TokenController(IRethinkDbStore store)
7    {
8        _store = store;
9    }
10
11    [HttpGet]
12    public Token Get()
13    {
14        var token = new Token
15        {
16            Id = Guid.NewGuid().ToString(),
17            Expires = DateTime.UtcNow.AddHours(1),
18            Issuer = Environment.MachineName
19        };
20
21        _store.InserToken(token);
22
23        return token;
24    }
25}
csharp

What we've done so far is registering an app instance as issuer at startup and persist generated tokens. Let's proceed further by creating a report with each issuer and the numbers of tokens generated.

First we create a IssuerStatus model to hold our report data:

1public class IssuerStatus
2{
3    public string Name { get; set; }
4    public string Version { get; set; }
5    public DateTime RegisterDate { get; set; }
6    public long TotalTokensIssued { get; set; }
7}
csharp

Then we we create a method in RethinkDbStore to build our report:

1public List<IssuerStatus> GetIssuerStatus()
2{
3    var conn = _connectionFactory.CreateConnection();
4    Cursor<Issuer> all = R.Db(_dbName).Table(nameof(Issuer)).RunCursor<Issuer>(conn);
5    var list = all.OrderByDescending(f => f.Timestamp)
6        .Select(f => new IssuerStatus
7        {
8            Name = f.Name,
9            RegisterDate = f.Timestamp,
10            Version = f.Version,
11            TotalTokensIssued = R.Db(_dbName).Table(nameof(Token))
12                    .GetAll(f.Name)[new { index = nameof(Token.Issuer) }]
13                    .Count()
14                    .Run<long>(conn)
15        }).ToList();
16
17    return list;
18}
csharp

Note are using the Token.Issuer secondary index to count all tokens belonging to the same issuer.

We expose this report over the TokenGen API by creating IssuerController:

1[Route("api/[controller]")]
2public class IssuerController : Controller
3{
4    private IRethinkDbStore _store;
5
6    public IssuerController(IRethinkDbStore store)
7    {
8        _store = store;
9    }
10
11    [HttpGet]
12    public dynamic Get()
13    {
14        return _store.GetIssuerStatus();
15    }
16}
csharp

Running the app service on Docker Swarm

Let's deploy the TokenGen app to docker swarm like we did in the previous guide. Create a Powershell script with the following content:

1# build tokengen image
2if(docker images -q tokengen-img){
3    "using existing tokengen image"
4}else{
5    docker build -t tokengen-img -f TokenGen.dockerfile .
6}
7
8# create and start tokengen service
9docker service create --publish 5000:5000 --name tokengen --network rdb-net tokengen-img
10
11# wait for the database initialization to finish
12Start-Sleep -s 10
13
14# scale x3
15docker service scale tokengen=3
powershell

Note that we are using the --network rdb-net param, so our app service will share the same network with the RethinkDB cluster. Before running the script we have to add a new RethinkDB connection data to the appsetting.json file. When our app runs inside docker swarm, the address of the RethinkDB proxy should be the name of the proxy service rdb-proxy. Best is to create a dedicated entry for the staging environment like this:

1"RethinkDbStaging": {
2"Host": "rdb-proxy",
3"Port": 28015,
4"Timeout": 10,
5"Database": "TokenStore"
6}
json

Then modify the Startup.cs to load these options if environment is not development:

1public void ConfigureServices(IServiceCollection services)
2{
3    //...
4
5    if (_env.IsDevelopment())
6    {
7        services.Configure<RethinkDbOptions>(Configuration.GetSection("RethinkDbDev"));
8    }
9    else
10    {
11        services.Configure<RethinkDbOptions>(Configuration.GetSection("RethinkDbStaging"));
12    }
13    services.AddSingleton<IRethinkDbConnectionFactory, RethinkDbConnectionFactory>();
14    services.AddSingleton<IRethinkDbStore, RethinkDbStore>();
15}
csharp

Note that we've set ENV ASPNETCORE_ENVIRONMENT="Staging" in TokenGen.dockerfile.

After running the deployment script use docker service ls command, you should get the following result:

1ID            NAME           REPLICAS  IMAGE             COMMAND
20184a1ou12ue  tokengen       3/3       tokengen-img
3157bd7yg7d60  rdb-secondary  2/2       rethinkdb:latest  rethinkdb --bind all --no-http-admin --join rdb-primary
441eloiad4jgp  rdb-primary    2/2       rethinkdb:latest  rethinkdb --bind all --no-http-admin --join rdb-secondary
567oci5m1wksi  rdb-proxy      1/1       rethinkdb:latest  rethinkdb proxy --bind all --join rdb-primary

If you access the RethinkDB web UI at http://localhost:8080 you can notice that the TokenStore database has been created and the Token and Issuer tables have 4 replicas each.

Testing the Docker Swarm load balancer

Create a load test for our cluster to see how Docker Swarm distributes the load between our app instances. For this purpose we will use a Powershell workflow that will call api/token endpoint in parallel, from 10 threads. After the load tests finishes, we'll call api/issuer endpoint to see how many tokens each app instance has generated.

1workflow loadtest{
2    Param($Iterations)
3
4    $array = 1..$Iterations
5
6    foreach -Parallel -ThrottleLimit 10 ($i in $array){
7        Invoke-RestMethod http://localhost:5000/api/token
8    }
9
10    Invoke-RestMethod http://localhost:5000/api/issuer
11}
12
13loadtest 500
powershell

This is the load test result on my PC:

1name              : d04aad77448a
2version           : 1.0.1.0
3registerDate      : 2016-08-17T16:53:40.321Z
4totalTokensIssued : 160
5
6name              : f6adf92273de
7version           : 1.0.1.0
8registerDate      : 2016-08-17T16:53:40.22Z
9totalTokensIssued : 161
10
11name              : cf3d85de52b4
12version           : 1.0.1.0
13registerDate      : 2016-08-17T16:53:31.62Z
14totalTokensIssued : 179

As you can see, the load balancer did a great job. We requested 500 tokens and each app container serviced a fair share of tokens.

Testing fault tolerance

We can simulate failures in our system by using the docker kill <container-id> command. First let's run docker ps to find out our containers ids:

1CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS
2d04aad77448a        tokengen-img:latest   "dotnet run"             2 hours ago         Up 2 hours
3f6adf92273de        tokengen-img:latest   "dotnet run"             2 hours ago         Up 2 hours
4cf3d85de52b4        tokengen-img:latest   "dotnet run"             2 hours ago         Up 2 hours
518d0548c1dae        rethinkdb:latest      "rethinkdb proxy --bi"   3 hours ago         Up 3 hours
67f14aa948092        rethinkdb:latest      "rethinkdb --bind all"   3 hours ago         Up 3 hours
7c38e7337d9f6        rethinkdb:latest      "rethinkdb --bind all"   3 hours ago         Up 3 hours
89fbbc3b72c18        rethinkdb:latest      "rethinkdb --bind all"   3 hours ago         Up 3 hours
9bb0c58d3c26d        rethinkdb:latest      "rethinkdb --bind all"   3 hours ago         Up 3 hours
docker

While the load test is running let's kill an app instance using docker kill d04aad77448a. If we run docker ps once more, we'll notice that the Docker Swarm engine has detected that one of the tokengen instance is down so he promptly started another one.

1CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS
2ec585e44960e        tokengen-img:latest   "dotnet run"             4 seconds ago       Up 1 seconds
3d04aad77448a        tokengen-img:latest   "dotnet run"             2 hours ago         Exited (137) 7 seconds ago
4f6adf92273de        tokengen-img:latest   "dotnet run"             2 hours ago         Up 2 hours
5cf3d85de52b4        tokengen-img:latest   "dotnet run"             2 hours ago         Up 2 hours
docker

As soon as the new instance is up, the load balancer will start distribute requests to it. You'll notice that id ec585e44960e is showing up in the issuers load test result.

Next let's see how RethinkDB cluster can cope with server failures. Open the RethinkDB web UI and go to the Token table details page. Locate the Servers used by this table section and look for the server name that holds the Primary replica. The server name should be in this format <container id>_<random letters>.

rdb-1

Now let's bring down the server that holds the primary replica with docker kill 7f14aa948092. As soon as the server becomes unavailable, the RethinkDB automatic fail-over kicks in and elects one of the remaining servers to act as primary replica.

rdb-2

In the same time, Docker Swarm has lunch a new RethinkDB server to replace the one that went down. If you look in the RethinkDB web UI server list, you'll notice that we still have 4 servers, but one of them doesn't hold any data.

rdb-3

In order to fully restore our cluster state, we have to reconfigure the replicas and expand to the new server. For this purpose I've created a RdbController in TokenGen app that calls RethinkDbStore.Reconfigure method:

1[Route("api/[controller]")]
2public class RdbController : Controller
3{
4    private IRethinkDbStore _store;
5
6    public RdbController(IRethinkDbStore store)
7    {
8        _store = store;
9    }
10
11    [Route("[action]/{shards:int}/{replicas:int}")]
12    public void Reconfigure(int shards, int replicas)
13    {
14        _store.Reconfigure(shards, replicas);
15    }
16}
csharp

We can now call the api/rdb/reconfigure from Powershell like this:

1Invoke-RestMethod http://localhost:5000/api/rdb/reconfigure/1/4
powershell

rdb-4

Reconfiguration is done and our cluster is healthy again.

Conclusion

Scalability and fault tolerance can be easily achieved nowadays if you use the right tools. For a production environment, you'll have to setup a DNS server, a reverse proxy and at least 3 Docker Swarm nodes to host the app and RethinkDB services. This will be the subject for another article.

The TokenGen project and all PowerShell scripts can be found on GitHub in this public repository.

Thanks for reading this guide.

You can also follow me on twitter @stefanprodan