Build a scalable, fault tolerant system with ASP.NET Core and RethinkDB on Docker Swarm mode
Introduction
This guide shows you how to use RethinkDB with ASP.NET Core and how to deploy your app along with a RethinkDB cluster using Docker for Windows and Docker Swarm mode. We will be using the TokenGen app from the Scale ASP.NET Core apps with Docker Swarm Mode guide and store the generated tokens along with the issuers in a RethinkDB database.
RethinkDB is an open source distributed document-oriented database. As most NoSQL databases, it stores JSON documents with no required schema or table structure.
What makes RethinkDB great:
- horizontal scaling
- table replication and sharing
- automatic fail-over
- query language with join support, map-reduce and geospatial queries
- pub/sub for data changes
If you want a scalable, fault tolerant database system for your ASP.NET Core applications then you should consider using RethinkDB.
Setting up a RethinkDB Cluster with Docker Swarm Mode
First we need to enable Swarm Mode and create a dedicated network for our RethinkDB cluster:
1# initialize swarm
2docker swarm init
3
4# create RethinkDB overlay network
5docker network create --driver overlay rdb-netWe start building our RethinkDB cluster by running a single RethinkDB server, we will remove this instance later on:
1# create and start rethinkdb primary
2docker service create --name rdb-primary --network rdb-net --replicas 1 rethinkdb:latest rethinkdb --bind all --no-http-adminNow we can create a secondary RethinkDB node that will join the rdb-primary node and form a cluster:
1# create and start rethinkdb secondary
2docker service create --name rdb-secondary --network rdb-net --replicas 1 rethinkdb:latest rethinkdb --bind all --no-http-admin --join rdb-primaryScale the secondary node so we can have a minimum of 3 nodes needed for RethinkDB automatic fail-over mechanism:
1# up 3 nodes (primary + two secondary) to enable automatic failover
2docker service scale rdb-secondary=2We now have a functional RethinkDB cluster, but we are not done yet.
Because we started the primary node without a join command, our cluster has a single point of failure.
If for some reason rdb-primary container crashes, the Docker Swarm engine will recreate and start this container, but he can't join the existing cluster. If we start new rdb-secondary instances, they will join the new rdb-primary container and form another cluster.
To resolve this issue we have to remove the rdb-primary service and recreate it with the join command like so:
1# remove primary
2docker service rm rdb-primary
3
4# recreate primary with --join flag
5docker service create --name rdb-primary --network rdb-net --replicas 1 rethinkdb:latest rethinkdb --bind all --no-http-admin --join rdb-secondaryNow we can also scale the primary node:
1# start two rdb-primary instances
2docker service scale rdb-primary=2At this point we have 4 nodes in our cluster, two rdb-primary and two rdb-secondary. We can further scale any of these two services and they will all join our cluster. If a rdb-primary or rdb-secondary instance crashes, the Docker Swarm will automatically start another container that will join our current cluster.
Last step is to create a RethinkDB proxy node, we expose port 8080 for the web admin and port 28015 so we can connect to the cluster from our app. Note that you don't have to expose port 28015 if your application runs inside the swarm. I'm exposing this port so I can connect to the cluster while debugging in VS.NET.
1# create and start rethinkdb proxy
2docker service create --name rdb-proxy --network rdb-net --publish 8080:8080 --publish 28015:28015 rethinkdb:latest rethinkdb proxy --bind all --join rdb-primaryOpen a browser and navigate to http://localhost:8080 to check the cluster state. In the servers page you should see 4 servers connected to the cluster.
if we run docker service ls we should get:
1ID NAME REPLICAS IMAGE COMMAND
2157bd7yg7d60 rdb-secondary 2/2 rethinkdb:latest rethinkdb --bind all --no-http-admin --join rdb-primary
341eloiad4jgp rdb-primary 2/2 rethinkdb:latest rethinkdb --bind all --no-http-admin --join rdb-secondary
467oci5m1wksi rdb-proxy 1/1 rethinkdb:latest rethinkdb proxy --bind all --join rdb-primaryConnecting to RethinkDB from ASP.NET Core
Open the TokenGen project from the previous guide and install the RethinkDB driver NuGet package:
1Install-Package RethinkDb.DriverWe are going to create an object that holds the RethinkDB cluster address. Add a class named RethinkDbOptions inside TokenGen project:
1public class RethinkDbOptions
2{
3 public string Host { get; set; }
4 public int Port { get; set; }
5 public string Database { get; set; }
6 public int Timeout { get; set; }
7}We will be storing the connection data in appconfig.json and use a RethinkDbOptions object to inject the data into RethinkDbConnectionFactory.
The RethinkDbConnectionFactory will provide a persistent connection to the RethinkDB cluster for our app:
1public class RethinkDbConnectionFactory : IRethinkDbConnectionFactory
2{
3 private static RethinkDB R = RethinkDB.R;
4 private Connection conn;
5 private RethinkDbOptions _options;
6
7 public RethinkDbConnectionFactory(IOptions<RethinkDbOptions> options)
8 {
9 _options = options.Value;
10 }
11
12 public Connection CreateConnection()
13 {
14 if (conn == null)
15 {
16 conn = R.Connection()
17 .Hostname(_options.Host)
18 .Port(_options.Port)
19 .Timeout(_options.Timeout)
20 .Connect();
21 }
22
23 if(!conn.Open)
24 {
25 conn.Reconnect();
26 }
27
28 return conn;
29 }
30
31 public void CloseConnection()
32 {
33 if (conn != null && conn.Open)
34 {
35 conn.Close(false);
36 }
37 }
38
39 public RethinkDbOptions GetOptions()
40 {
41 return _options;
42 }
43}Open appsettings.json and add the RethinkDB cluster connection data:
1 "RethinkDbDev": {
2 "Host": "localhost",
3 "Port": 28015,
4 "Timeout": 10,
5 "Database": "TokenStore"
6 }Now we can create a singleton instance of RethinkDbConnectionFactory in Startup.cs. You should reuse the same connection across whole application since the RethinkDb connection is thread safe.
1public void ConfigureServices(IServiceCollection services)
2{
3 //....
4
5 services.Configure<RethinkDbOptions>(Configuration.GetSection("RethinkDbDev"));
6 services.AddSingleton<IRethinkDbConnectionFactory, RethinkDbConnectionFactory>();
7}Test the connection inside Startup.Configure method:
1public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory,
2 IRethinkDbConnectionFactory connectionFactory)
3{
4 //....
5
6 var con = connectionFactory.CreateConnection();
7 con.CheckOpen();
8}Database initialization
The TokenGen app logic involves two entities: token and issuer. Tokens are issued by an app instance (issuer) to a client by calling the /api/token endpoint. We will create a database to store these two entries.
First we create a class for each entity:
1public class Issuer
2{
3 [JsonProperty("id", NullValueHandling = NullValueHandling.Ignore)]
4 public string Id { get; set; }
5 public string Name { get; set; }
6 public string Version { get; set; }
7 public DateTime Timestamp { get; set; }
8}
9
10public class Token
11{
12 [JsonProperty("id", NullValueHandling = NullValueHandling.Ignore)]
13 public string Id { get; set; }
14 public DateTime Expires { get; set; }
15 public string Issuer { get; set; }
16}Now we will create a service that uses the RethinkDb connection factory and implements the database initialization logic:
1public class RethinkDbStore : IRethinkDbStore
2{
3 private static IRethinkDbConnectionFactory _connectionFactory;
4 private static RethinkDB R = RethinkDB.R;
5 private string _dbName;
6
7 public RethinkDbStore(IRethinkDbConnectionFactory connectionFactory)
8 {
9 _connectionFactory = connectionFactory;
10 _dbName = connectionFactory.GetOptions().Database;
11 }
12
13 public void InitializeDatabase()
14 {
15 // database
16 CreateDb(_dbName);
17
18 // tables
19 CreateTable(_dbName, nameof(Token));
20 CreateTable(_dbName, nameof(Issuer));
21
22 // indexes
23 CreateIndex(_dbName, nameof(Token), nameof(Token.Issuer));
24 CreateIndex(_dbName, nameof(Issuer), nameof(Issuer.Name));
25
26 }
27
28 protected void CreateDb(string dbName)
29 {
30 var conn = _connectionFactory.CreateConnection();
31 var exists = R.DbList().Contains(db => db == dbName).Run(conn);
32
33 if (!exists)
34 {
35 R.DbCreate(dbName).Run(conn);
36 R.Db(dbName).Wait_().Run(conn);
37 }
38 }
39
40 protected void CreateTable(string dbName, string tableName)
41 {
42 var conn = _connectionFactory.CreateConnection();
43 var exists = R.Db(dbName).TableList().Contains(t => t == tableName).Run(conn);
44 if (!exists)
45 {
46 R.Db(dbName).TableCreate(tableName).Run(conn);
47 R.Db(dbName).Table(tableName).Wait_().Run(conn);
48 }
49 }
50
51 protected void CreateIndex(string dbName, string tableName, string indexName)
52 {
53 var conn = _connectionFactory.CreateConnection();
54 var exists = R.Db(dbName).Table(tableName).IndexList().Contains(t => t == indexName).Run(conn);
55 if (!exists)
56 {
57 R.Db(dbName).Table(tableName).IndexCreate(indexName).Run(conn);
58 R.Db(dbName).Table(tableName).IndexWait(indexName).Run(conn);
59 }
60 }
61
62 public void Reconfigure(int shards, int replicas)
63 {
64 var conn = _connectionFactory.CreateConnection();
65 var tables = R.Db(_dbName).TableList().Run(conn);
66 foreach (string table in tables)
67 {
68 R.Db(_dbName).Table(table).Reconfigure().OptArg("shards", shards).OptArg("replicas", replicas).Run(conn);
69 R.Db(_dbName).Table(table).Wait_().Run(conn);
70 }
71 }
72}The InitializeDatabase method checks if the TokenStore database, tables and indexes are present on the cluster, if they are missing then it will create them. Because these operations are expensive you should call this method only once at application start.
On application startup, create a singleton instance of IRethinkDbStore and invoke InitializeDatabase:
1public void ConfigureServices(IServiceCollection services)
2{
3 //...
4
5 services.Configure<RethinkDbOptions>(Configuration.GetSection("RethinkDbDev"));
6 services.AddSingleton<IRethinkDbConnectionFactory, RethinkDbConnectionFactory>();
7
8 services.AddSingleton<IRethinkDbStore, RethinkDbStore>();
9}
10
11public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory,
12 IRethinkDbStore store)
13{
14 //...
15
16 store.InitializeDatabase();
17}When the InitializeDatabase runs, the tables are created by default with one shard and one replica. We've setup earlier a RethinkDB cluster with 4 nodes, to make our tables fault tolerant we need to replicate our data on all 4 nodes. In order to do this, after the database initialization we can call store.Reconfigure(shards = 1, replicas = 4).
CRUD Operations
When a TokenGen app instance starts it has to register in the database as an issuer, we will use the MachineName to identity the issuer. When docker starts a new container, the MachineName gets populated with the container ID, so we will use this ID to track the load of each app instance.
First we create a method to persist issuers:
1public class RethinkDbStore : IRethinkDbStore
2{
3 //..
4
5 public string InsertOrUpdateIssuer(Issuer issuer)
6 {
7 var conn = _connectionFactory.CreateConnection();
8 Cursor<Issuer> all = R.Db(_dbName).Table(nameof(Issuer))
9 .GetAll(issuer.Name)[new { index = nameof(Issuer.Name) }]
10 .Run<Issuer>(conn);
11
12 var issuers = all.ToList();
13
14 if (issuers.Count > 0)
15 {
16 // update
17 R.Db(_dbName).Table(nameof(Issuer)).Get(issuers.First().Id).Update(issuer).RunResult(conn);
18
19 return issuers.First().Id;
20 }
21 else
22 {
23 // insert
24 var result = R.Db(_dbName).Table(nameof(Issuer))
25 .Insert(issuer)
26 .RunResult(conn);
27
28 return result.GeneratedKeys.First().ToString();
29 }
30 }
31}This method uses the Issuer.Name secondary index we created earlier to search for an issuer by name. If the issuer exists, it will update the version and timestamp, else it will insert a new one.
Note that the update will only happen when you run the app from Visual Studio. On docker swarm, when you scale up a service, each instance has a unique ID.
We can now call this method on application start:
1public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory,
2 IRethinkDbStore store)
3{
4 //...
5
6 // create TokenStore database, tables and indexes if not exists
7 store.InitializeDatabase();
8
9 // register issuer
10 store.InsertOrUpdateIssuer(new Issuer
11 {
12 Name = Environment.MachineName,
13 Version = PlatformServices.Default.Application.ApplicationVersion,
14 Timestamp = DateTime.UtcNow
15 });
16}Now that we've implemented the issuers persistence we can do the same for the tokens by adding the following method to RethinkDbStore:
1public void InserToken(Token token)
2{
3 var conn = _connectionFactory.CreateConnection();
4 var result = R.Db(_dbName).Table(nameof(Token))
5 .Insert(token)
6 .RunResult(conn);
7}Edit the TokenController and call InsertToken before sending the token to the client:
1[Route("api/[controller]")]
2public class TokenController : Controller
3{
4 private IRethinkDbStore _store;
5
6 public TokenController(IRethinkDbStore store)
7 {
8 _store = store;
9 }
10
11 [HttpGet]
12 public Token Get()
13 {
14 var token = new Token
15 {
16 Id = Guid.NewGuid().ToString(),
17 Expires = DateTime.UtcNow.AddHours(1),
18 Issuer = Environment.MachineName
19 };
20
21 _store.InserToken(token);
22
23 return token;
24 }
25}What we've done so far is registering an app instance as issuer at startup and persist generated tokens. Let's proceed further by creating a report with each issuer and the numbers of tokens generated.
First we create a IssuerStatus model to hold our report data:
1public class IssuerStatus
2{
3 public string Name { get; set; }
4 public string Version { get; set; }
5 public DateTime RegisterDate { get; set; }
6 public long TotalTokensIssued { get; set; }
7}Then we we create a method in RethinkDbStore to build our report:
1public List<IssuerStatus> GetIssuerStatus()
2{
3 var conn = _connectionFactory.CreateConnection();
4 Cursor<Issuer> all = R.Db(_dbName).Table(nameof(Issuer)).RunCursor<Issuer>(conn);
5 var list = all.OrderByDescending(f => f.Timestamp)
6 .Select(f => new IssuerStatus
7 {
8 Name = f.Name,
9 RegisterDate = f.Timestamp,
10 Version = f.Version,
11 TotalTokensIssued = R.Db(_dbName).Table(nameof(Token))
12 .GetAll(f.Name)[new { index = nameof(Token.Issuer) }]
13 .Count()
14 .Run<long>(conn)
15 }).ToList();
16
17 return list;
18}Note are using the Token.Issuer secondary index to count all tokens belonging to the same issuer.
We expose this report over the TokenGen API by creating IssuerController:
1[Route("api/[controller]")]
2public class IssuerController : Controller
3{
4 private IRethinkDbStore _store;
5
6 public IssuerController(IRethinkDbStore store)
7 {
8 _store = store;
9 }
10
11 [HttpGet]
12 public dynamic Get()
13 {
14 return _store.GetIssuerStatus();
15 }
16}Running the app service on Docker Swarm
Let's deploy the TokenGen app to docker swarm like we did in the previous guide. Create a Powershell script with the following content:
1# build tokengen image
2if(docker images -q tokengen-img){
3 "using existing tokengen image"
4}else{
5 docker build -t tokengen-img -f TokenGen.dockerfile .
6}
7
8# create and start tokengen service
9docker service create --publish 5000:5000 --name tokengen --network rdb-net tokengen-img
10
11# wait for the database initialization to finish
12Start-Sleep -s 10
13
14# scale x3
15docker service scale tokengen=3Note that we are using the --network rdb-net param, so our app service will share the same network with the RethinkDB cluster. Before running the script we have to add a new RethinkDB connection data to the appsetting.json file.
When our app runs inside docker swarm, the address of the RethinkDB proxy should be the name of the proxy service rdb-proxy.
Best is to create a dedicated entry for the staging environment like this:
1"RethinkDbStaging": {
2"Host": "rdb-proxy",
3"Port": 28015,
4"Timeout": 10,
5"Database": "TokenStore"
6}Then modify the Startup.cs to load these options if environment is not development:
1public void ConfigureServices(IServiceCollection services)
2{
3 //...
4
5 if (_env.IsDevelopment())
6 {
7 services.Configure<RethinkDbOptions>(Configuration.GetSection("RethinkDbDev"));
8 }
9 else
10 {
11 services.Configure<RethinkDbOptions>(Configuration.GetSection("RethinkDbStaging"));
12 }
13 services.AddSingleton<IRethinkDbConnectionFactory, RethinkDbConnectionFactory>();
14 services.AddSingleton<IRethinkDbStore, RethinkDbStore>();
15}Note that we've set ENV ASPNETCORE_ENVIRONMENT="Staging" in TokenGen.dockerfile.
After running the deployment script use docker service ls command, you should get the following result:
1ID NAME REPLICAS IMAGE COMMAND
20184a1ou12ue tokengen 3/3 tokengen-img
3157bd7yg7d60 rdb-secondary 2/2 rethinkdb:latest rethinkdb --bind all --no-http-admin --join rdb-primary
441eloiad4jgp rdb-primary 2/2 rethinkdb:latest rethinkdb --bind all --no-http-admin --join rdb-secondary
567oci5m1wksi rdb-proxy 1/1 rethinkdb:latest rethinkdb proxy --bind all --join rdb-primaryIf you access the RethinkDB web UI at http://localhost:8080 you can notice that the TokenStore database has been created and the Token and Issuer tables have 4 replicas each.
Testing the Docker Swarm load balancer
Create a load test for our cluster to see how Docker Swarm distributes the load between our app instances. For this purpose we will use a Powershell workflow that will call api/token endpoint in parallel, from 10 threads. After the load tests finishes, we'll call api/issuer endpoint to see how many tokens each app instance has generated.
1workflow loadtest{
2 Param($Iterations)
3
4 $array = 1..$Iterations
5
6 foreach -Parallel -ThrottleLimit 10 ($i in $array){
7 Invoke-RestMethod http://localhost:5000/api/token
8 }
9
10 Invoke-RestMethod http://localhost:5000/api/issuer
11}
12
13loadtest 500This is the load test result on my PC:
1name : d04aad77448a
2version : 1.0.1.0
3registerDate : 2016-08-17T16:53:40.321Z
4totalTokensIssued : 160
5
6name : f6adf92273de
7version : 1.0.1.0
8registerDate : 2016-08-17T16:53:40.22Z
9totalTokensIssued : 161
10
11name : cf3d85de52b4
12version : 1.0.1.0
13registerDate : 2016-08-17T16:53:31.62Z
14totalTokensIssued : 179As you can see, the load balancer did a great job. We requested 500 tokens and each app container serviced a fair share of tokens.
Testing fault tolerance
We can simulate failures in our system by using the docker kill <container-id> command. First let's run docker ps to find out our containers ids:
1CONTAINER ID IMAGE COMMAND CREATED STATUS
2d04aad77448a tokengen-img:latest "dotnet run" 2 hours ago Up 2 hours
3f6adf92273de tokengen-img:latest "dotnet run" 2 hours ago Up 2 hours
4cf3d85de52b4 tokengen-img:latest "dotnet run" 2 hours ago Up 2 hours
518d0548c1dae rethinkdb:latest "rethinkdb proxy --bi" 3 hours ago Up 3 hours
67f14aa948092 rethinkdb:latest "rethinkdb --bind all" 3 hours ago Up 3 hours
7c38e7337d9f6 rethinkdb:latest "rethinkdb --bind all" 3 hours ago Up 3 hours
89fbbc3b72c18 rethinkdb:latest "rethinkdb --bind all" 3 hours ago Up 3 hours
9bb0c58d3c26d rethinkdb:latest "rethinkdb --bind all" 3 hours ago Up 3 hoursWhile the load test is running let's kill an app instance using docker kill d04aad77448a.
If we run docker ps once more, we'll notice that the Docker Swarm engine has detected that one of the tokengen instance is down so he promptly started another one.
1CONTAINER ID IMAGE COMMAND CREATED STATUS
2ec585e44960e tokengen-img:latest "dotnet run" 4 seconds ago Up 1 seconds
3d04aad77448a tokengen-img:latest "dotnet run" 2 hours ago Exited (137) 7 seconds ago
4f6adf92273de tokengen-img:latest "dotnet run" 2 hours ago Up 2 hours
5cf3d85de52b4 tokengen-img:latest "dotnet run" 2 hours ago Up 2 hoursAs soon as the new instance is up, the load balancer will start distribute requests to it. You'll notice that id ec585e44960e is showing up in the issuers load test result.
Next let's see how RethinkDB cluster can cope with server failures. Open the RethinkDB web UI and go to the Token table details page. Locate the Servers used by this table section and look for the server name that holds the Primary replica. The server name should be in this format <container id>_<random letters>.
Now let's bring down the server that holds the primary replica with docker kill 7f14aa948092. As soon as the server becomes unavailable, the RethinkDB automatic fail-over kicks in and elects one of the remaining servers to act as primary replica.
In the same time, Docker Swarm has lunch a new RethinkDB server to replace the one that went down. If you look in the RethinkDB web UI server list, you'll notice that we still have 4 servers, but one of them doesn't hold any data.
In order to fully restore our cluster state, we have to reconfigure the replicas and expand to the new server.
For this purpose I've created a RdbController in TokenGen app that calls RethinkDbStore.Reconfigure method:
1[Route("api/[controller]")]
2public class RdbController : Controller
3{
4 private IRethinkDbStore _store;
5
6 public RdbController(IRethinkDbStore store)
7 {
8 _store = store;
9 }
10
11 [Route("[action]/{shards:int}/{replicas:int}")]
12 public void Reconfigure(int shards, int replicas)
13 {
14 _store.Reconfigure(shards, replicas);
15 }
16}We can now call the api/rdb/reconfigure from Powershell like this:
1Invoke-RestMethod http://localhost:5000/api/rdb/reconfigure/1/4
Reconfiguration is done and our cluster is healthy again.
Conclusion
Scalability and fault tolerance can be easily achieved nowadays if you use the right tools. For a production environment, you'll have to setup a DNS server, a reverse proxy and at least 3 Docker Swarm nodes to host the app and RethinkDB services. This will be the subject for another article.
The TokenGen project and all PowerShell scripts can be found on GitHub in this public repository.
Thanks for reading this guide.
You can also follow me on twitter @stefanprodan