Handling File Upload Using Ruby on Rails 5 API

Sending basic JSON data made out of strings to an API (Application Programming Interface) is, in most cases, a simple and straightforward task. But what about sending files that consist of numerous lines of binary data in various formats? Such data require a slightly different approach for sending files to the API.

In this guide we will examine the two main approaches of handling file uploads, multipart form data and base64 encoding, through a Rails 5 API application using both the paperclip and the carrierwave gems.

Multipart forms have been around since HTML 4. They were introduced because the standard application/x-www-form-urlencoded forms did not handle bigger amounts of data sufficiently well.

According to W3C, multipart/form-data is used for forms that "contain files, non-ASCII data, and binary data." What multipart forms do differently is that they break the data into different chunks representing the various characteristics of the file (size, content type, name, contents) and the characteristics of the form's standard text and number data.

To make things clearer, let's consider the following form:

If the user enters "John" in the text input, and selects the text file "file1.txt", the browser will send back the following data:

Every part of the form is separated by a boundary, which represents a different string (AaB03x in the example). The part itself contains binary infomation with a Content-Type. Larger files can be broken down in chuncks and assembled in the server, enabling a file to be streamed and to have its integrity maintaned in cases of connection interruprtion.

Let's consider another file. If the user selects another file "file2.gif", the browser will construct the parts as follows:

Here, it can be seen that there is another part added to the form-data. This time, since the file is not in a text/plain format, it is broken down in binary, as it can be inferred from the Content-Transfer-Encoding property.

The Content-Type property gives information about the type of the file, also known as its media (MIME) type. If the Content-Type property is not defined and the file is not in text format, its format will default to application/octet-stream , which means that the final file will be binary and without a type.

When sending data to an API, it is always good to include a Content-Type to each part which contains a file, otherwise there is no way to validate the contents of the file.

Base64 for is one of the most commonly used binary to text encoding formats. It uses an algorithm to break up binary code in pieces and convert it in ASCII characters (text). It has a wide array of applications - apart from being used to encode files into text in order to send them to an API, it is also used represent images as a content source in CSS, HTML and SVG.

The structure of base64 encoded files is very simple. It consists of two parts - a MIME type (similar to the multipart Content-Type and the actual base64 encoded string:

When it comes to uploading, the file is encoded into base64 on the client and decoded on the server. The base64 string can be easily attached to a JSON object's attribute:

An API would easily be able to pick up the parameter and decode it back to binary. This makes base64-encoded file uploads convenient for APIs, since the format of the message containing the file and the way it is transferred does not differ from the standard way information is sent to an API.

Base64-encoded files can be easily used by JSON and XML APIs since they are represented as text and can be sent through the standard application/json format. However, the encoding increases the file size by 33%, making it difficult to transfer larger files. Encoding and decoding also adds a computational overhead for both the server and the client. Therefore, base64 is suitable for sending images and small files under 100MB.

Multipart forms, on the other hand, are more "unnatural" to the APIs, since the data is encoded in a different format and requires a different way of handling. However, the increased performance with larger files and the ability to stream files makes multipart form data more desirable for uploading larger files, such as videos.

Let's apply these concepts. First, let's set up a Rails 5 API application and scaffold a model that is going to be used.

To create a Rails 5 API, you need Ruby 2.2.4 installed. After you have a suitable Ruby version, the first step is to install the newest version of Rails through your terminal/command prompt:

This will give you the ability to run rails new using the most recent version (currently 5.0.0.beta3):

This will create a brand new Rails 5 app, and with the --api option included in the generator, it will be all set up to be used as an API. Move to the directory of the new project:

Go to the Gemfile and uncomment jbuilder and rack-cors .

JBuilder is used to create JSON structures for the responses from the application. In MVC terms, the JSON responses are going to be the view layer (AKA UI layer or presentation layer) of the application. Therefore, all Jbuilder-generated responses have to be put in app/views/(view for a particular controller action) .

rack-cors enables cross-origin resource sharing (CORS). Simply put, rack-cors will enable browser-based applications (AngularJS, React) and mobile applications to request information from the API. Go to the application configuration file and add the configurations for CORS:

This configuration will give full access to the API (* means that everything is accepted). This is not a problem for this guide, since we are going to work only with a local server.

Install the gems:

With the gems installed, the model is going to be scaffolded with Jbuilder-generated views and be ready to be consumed by client-side applications.

A model named Item is going to be scaffolded which has a name and description as strings. Note that there is still no information about the file - file information will be included in the model's schema at a later stage. For now, we are all set to go ahead and migrate the model into the database:

This will create a table in the database for the new model. Note that in Rails 5 you can use rails instead of rake for executing a migration command.

To continue with the guide, you have to choose the gem with which you would like to implement your file uploads. If you would like to do both, I recommended that you use version control such as git so that the implementations don't clash.

To get started with Paperclip, first add the gem to your Gemfile:

And install it:

First, generate a migration that will add the attachment to the database. You have to put the same name for the attachment in the model as the one you put in the generator (in this case, it is picture).

After you are done, don't forget to migrate the database:

Second, add the following lines in the file of the model:

Here is what each method is about:

Third, add a permitted parameter that is going to accept :picture:

The last step is to add the :picture to the Jbuilder view, so that when we GET an item, it will return its picture:

To make uploading multiple files using Paperclip possible, add another model that is related to the main model and that will contain the files. In the current example, let's add the capability for an item to have multiple PDF documents. First, let's generate a model:

The first generator is going to produce a model named document with a name as a string. item:references will generate a belongs_to :item reference which means that the item and the document will have a one-to-many relationship. The second generator will generate fields for the attachment.

A reference also has to be added to the Item model as well:

Migrate the changes to the database schema:

With this step finished, the document model is ready to be configured. Let's add the necessarry methods and validations to add PDF documents.

This time, validates attachment checks if the document's type is application/pdf .

With this addition, the document model is ready. Next, things need to be handled in the controller in order to handle creation of multiple files.

When a new item is created, another parameter must get sent. document_data will contain an array of data about each document, either in multipart or in base64 format:

Add the logic for creating multiple files in the controller:

In order to get :document_data into the model, a attr_accessor method needs to be added to the model:

After the item is successfully saved, the document_data parameter has to be checked. If the document data array is not empty, the controller will call the save_attachments that will attempt to create the documents. Here is how the model method has to look like:

The save_attachments method is going to go through all the items in the document_data array. For each document, there is going to be an object from the document model created.

All done! Now it is possible to add multiple files to a document. Add :documents to the Jbuilder view so that the documents get returned together with the item.

Suppose you want to create items and documents by sending in base64-encoded strings instead of form data. There are several additional steps that should be done for this to be achieved. As mentioned before, base64 involves much more overhead computation work and much more memory, but it is more easily integrated into the API.

First, the attr_accessor method will be used to get the base64 encoded string in the model:

Second, add a private method for decoding image_base and assigning it to the picture attribute of the item.

The parse_image method takes image_base and puts it into Paperclip's IO adapters. They contain a registry which can turn the base64 string into binary. Because the file name is not stored, you can either put an arbitrary value (like "file.jpg", even if your file is not in jpg format) or add another attr_accessor for the name itself. Finally, self.picture = image assigns the image to the current instance of the object.

The method is ready, but it has to be called every time a new object is created. Let's add a filter that will call the parse_image method when that happens:

The before_validation method will ensure that the base64 string will be parsed and assigned to the object instance before Paperclip validates the size, content type, and the rest of the attributes of the file.

In order to get image_base, it has to be entered as a parameter in the controller:

Similarly to the item, the document data has to be decoded before being saved. So instead of sending a :file directly, the :file_contents have to be sent:

Similarly to the item model, add attr_accessor for the :file_contents and a method to decode the base64-encoded content:

Add the Carrierwave gem to your Gemfile:

Run the installation:

CarrierWave isolates the logic of the uploaded files from the logic of the models using uploaders. To generate an uploader for the picture of the Item model, run the following command:

The first generator will create a new directory in the application - app/uploaders . This is where all the uploaders are going to be stored. The second generator will add a picture column as string in the Items table which will store a reference to the file.

Don't forget to migrate the database:

In the generated uploader, you can see various configuration options. Right now we need the file type validation option. Since the uploader is used to upload pictures, the content types of each picture need to be whitelisted (or approved):

Now, mount the uploader to the item model:

When a new model instance is created, the uploader will automatically associate the picture with it. The picture will contain the url with the image, which can be reached through Item.picture.url.

Next, enter the :picture parameter as a parameter:

Last, add the :picture to the Jbuilder view, so that when we GET an item, the API will return its picture:

Let's add PDF documents to the item. This will require adding another uploader as well as a model named document belonging to the item. To accomplish this task, run these two generators:

The first generator creates the document model with references to the Item and to a file column that gives a reference to its CarrierWave file. The second generator creates the uploader in the app/uploaders directory. Add the new document model in the database schema:

Mount the uploader in the document model:

Because the documents will be PDF only, adjust the whitelist accordingly in the uploader:

In the Item model, add the following lines:

has_many :documents adds a relation one-to-many between the item and the documents. attr_accessor :document_data will let us send extra attributes to the controller. This attribute is going to contain an array with data about every PDF document.

We are done with the model! Let's proceed to the controllers:

Finally, update your Jbuilder view, so that the documents get returned together with the item:

Uploading base64-encoded files is a very easy task with Carrierwave. Just add the carrierwave-base64 gem:

Install it:

Go to your models and replace mount_uploader with mount_base64_uploader

And voila, your application can now accept base64-encoded files. It is as simple as that.

For POSt-ing multipart form data, cURL is convenient enough. Keep in mind that the paths included are arbitrary and you have to add your own. Another thing - when sending files, make sure to declare their type (type=application/pdf). Otherwise, cURL sends files as a binary type ( application/octet-stream ); this may break some of the validations.

If you want to send base64 data, keep in mind that the strings are long, and cURL might be tedious to use. I would recommend Postman . Note: Replace "picture" with " image_base" if you used Paperclip. Here is the JSON format that the API is going to accept:

Now you know how to upload files. You can now go ahead and set up a client-side application that can consume the API. In case you got lost in the guide, I created a Github repository and added branches for each of the steps and the implementations done in the tutorial.

My personal preference for file upload is CarrieWave because it offers a more straightforward approach. The logic for the file is isolated into uploaders, which can be reused repeatedly for multiple models. This makes the code more maintinable. It took me considerably less time to implement file uploading for both single and multiple items. With the CarrierWave-base64 gem, I did not have to add any custom logic for decoding and saving images - changing the name of the mounting method was enough. The only significant drawback is that CarrierWave has a lot of "magic" going behind it. It does not feel as customizable as Paperclip, and debugging on CarrierWave seems more difficult.

On the other hand, Paperclip offers a more personalized approach to file uploads. All the logic is stored inside the model, and there is no need to encapsulate the logic outside the model. Files are treated more like standard fields in the model rather than independent parts that are merely associated with the model. Debugging is somewhat intuitive, since any errors are simply treated as attribute validation errors. On the other hand, making the image uploading more generic requires more tinkering and experience (putting methods in ApplicationRecord is one way). Moreover, Base64 uploads do not come as naturally as they do for CarrierWave, so there is a need for custom logic.

To conclude, both CarrierWave and Paperclip have their pros and cons, and the choice of which to use depends on the case. Both gems are popular and have big communities built around them. The same applies for the different ways of uploading files - both base64 and multipart forms are widely-used means of uploading files to APIs. The choice depends on the scope and requirements of the project you are working on.