File Controller and Finishing a simple File Storage Service Using VueJS, Flask, and RethinkDB
Getting Started
For more information about how to set up your workspace, checkout the first guide in this series: Introduction and Setup to Building a Simple File Storage Service Using VueJS, Flask and RethinkDB.
File Controller
The File controllers will be used for working with both Files and Folders, hence, there will be slightly more logic here than in our previous controller. We start by creating a boilerplate for our controller in /api/controllers/files.py module.
1import os
2
3from flask import request, g
4from flask_restful import reqparse, abort, Resource
5from werkzeug import secure_filename
6
7from api.models import File
8
9BASE_DIR = os.path.abspath(
10 os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
11)
12
13
14class CreateList(Resource):
15 def get(self, user_id):
16 pass
17
18 def post(self, user_id):
19 pass
20
21class ViewEditDelete(Resource):
22 def get(self, user_id, file_id):
23 pass
24
25 def put(self, user_id, file_id):
26 pass
27
28 def delete(self, user_id, file_id):
29 passThe CreateList class, as the name implies, will be used for creating and listing files for a logged in user. The ViewEditDelete class, also as the name implies, will be used for viewing, editing, and deleting files. The methods we're using in the classes correspond with the appropriate HTTP actions.
Decorators
We will start the implementation by creating a bunch of decorators which we will be using on the methods in our Resource classes. You'll want to separate this out into the /api/utils/decorators.py module.
1from jose import jwt
2from jose.exceptions import JWTError
3from functools import wraps
4
5from flask import current_app, request, g
6from flask_restful import abort
7
8from api.models import User, File
9
10def login_required(f):
11 '''
12 This decorator checks the header to ensure a valid token is set
13 '''
14 @wraps(f)
15 def func(*args, **kwargs):
16 try:
17 if 'authorization' not in request.headers:
18 abort(404, message="You need to be logged in to access this resource")
19 token = request.headers.get('authorization')
20 payload = jwt.decode(token, current_app.config['SECRET_KEY'], algorithms=['HS256'])
21 user_id = payload['id']
22 g.user = User.find(user_id)
23 if g.user is None:
24 abort(404, message="The user id is invalid")
25 return f(*args, **kwargs)
26 except JWTError as e:
27 abort(400, message="There was a problem while trying to parse your token -> {}".format(e.message))
28 return func
29
30def validate_user(f):
31 '''
32 This decorate ensures that the user logged in is the actually the same user we're operating on
33 '''
34 @wraps(f)
35 def func(*args, **kwargs):
36 user_id = kwargs.get('user_id')
37 if user_id != g.user['id']:
38 abort(404, message="You do not have permission to the resource you are trying to access")
39 return f(*args, **kwargs)
40 return func
41
42def belongs_to_user(f):
43 '''
44 This decorator ensures that the file we're trying to access actually belongs to us
45 '''
46 @wraps(f)
47 def func(*args, **kwargs):
48 file_id = kwargs.get('file_id')
49 user_id = kwargs.get('user_id')
50 file = File.find(file_id, True)
51 if not file or file['creator'] != user_id:
52 abort(404, message="The file you are trying to access was not found")
53 g.file = file
54 return f(*args, **kwargs)
55 return funcThe login_required decorator is used to validate that users are actually logged in before accessing the method's functionality. We use this decorator to protect certain endpoints by decoding the token to ensure its validity. We get the id field stored in the token and try to retrieve the corresponding user object. Then, also store this object in g.user for access within the method definition.
Similarly, we create the validate_user decorator which ensures that no other logged in user can access URL patterns labelled with another user's ID. This validation is purely based on the information in the URL.
Finally, the belongs_to_user decorator ensures that only the user who created a file can access it. This decorator actually checks the creator field in the file document against the user_id supplied.
Here are the views for creation of new files and listing of files:
1class CreateList(Resource):
2 @login_required
3 @validate_user
4 @marshal_with(file_array_serializer)
5 def get(self, user_id):
6 try:
7 return File.filter({'creator': user_id, 'parent_id': '0'})
8 except Exception as e:
9 abort(500, message="There was an error while trying to get your files --> {}".format(e.message))
10
11 @login_required
12 @validate_user
13 @marshal_with(file_serializer)
14 def post(self, user_id):
15 try:
16 parser = reqparse.RequestParser()
17 parser.add_argument('name', type=str, help="This should be the folder name if creating a folder")
18 parser.add_argument('parent_id', type=str, help='This should be the parent folder id')
19 parser.add_argument('is_folder', type=bool, help="This indicates whether you are trying to create a folder or not")
20
21 args = parser.parse_args()
22
23 name = args.get('name', None)
24 parent_id = args.get('parent_id', None)
25 is_folder = args.get('is_folder', False)
26
27 parent = None
28
29 # Are we adding this to a parent folder?
30 if parent_id is not None:
31 parent = File.find(parent_id)
32 if parent is None:
33 raise Exception("This folder does not exist")
34 if not parent['is_folder']:
35 raise Exception("Select a valid folder to upload to")
36
37 # Are we creating a folder?
38 if is_folder:
39 if name is None:
40 raise Exception("You need to specify a name for this folder")
41
42 return Folder.create(
43 name=name,
44 parent=parent,
45 is_folder=is_folder,
46 creator=user_id
47 )
48 else:
49 files = request.files['file']
50
51 if files and is_allowed(files.filename):
52 _dir = os.path.join(BASE_DIR, 'upload/{}/'.format(user_id))
53
54 if not os.path.isdir(_dir):
55 os.mkdir(_dir)
56 filename = secure_filename(files.filename)
57 to_path = os.path.join(_dir, filename)
58 files.save(to_path)
59 fileuri = os.path.join('upload/{}/'.format(user_id), filename)
60 filesize = os.path.getsize(to_path)
61
62 return File.create(
63 name=filename,
64 uri=fileuri,
65 size=filesize,
66 parent=parent,
67 creator=user_id
68 )
69 raise Exception("You did not supply a valid file in your request")
70 except Exception as e:
71 abort(500, message="There was an error while processing your request --> {}".format(e.message))The listing method is pretty straightforward. We filter the table for all files that were created by a certain user and stored in the root directory. We return this data for this endpoint and throw an exception if there are any errors.
Create
For the create action, it's a bit more involved. For this guide, we're assuming that files and folders will be created with the same endpoint. For files, we will need to supply the file as well as a parent_id, if we're uploading it in a folder. For folders, we will need a name and a parent_id value, again if we are creating this within another folder. For folders, we also need to send an is_folder field with our request to specify that we are creating a folder.
If we are going to store this within a folder, we have to ensure that the folder exists and is a valid folder. We also ensure that we are supplying a name field if we are creating a folder.
For file creation, we upload the file into a folder specifically named for the different users as mentioned before. In our case, we are using the pattern /upload/<user_id> for the different user file directories. We have also used the file information to populate the document we're going to be storing in the table.
We conclude by calling the methods for file and folder creation - File.create() and Folder.create() respectively.
Serializers
Notice that we have used the marshal_with decorator available with Flask-RESTful. This decorator is used to format the response object and to indicate the different field names and types that we'll be returning. See the definition of the file_array_serializer and file_serializer below:
1file_array_serializer = {
2 'id': fields.String,
3 'name': fields.String,
4 'size': fields.Integer,
5 'uri': fields.String,
6 'is_folder': fields.Boolean,
7 'parent_id': fields.String,
8 'creator': fields.String,
9 'date_created': fields.DateTime(dt_format= 'rfc822'),
10 'date_modified': fields.DateTime(dt_format='rfc822'),
11}
12
13file_serializer = {
14 'id': fields.String,
15 'name': fields.String,
16 'size': fields.Integer,
17 'uri': fields.String,
18 'is_folder': fields.Boolean,
19 'objects': fields.Nested(file_array_serializer, default=[]),
20 'parent_id': fields.String,
21 'creator': fields.String,
22 'date_created': fields.DateTime(dt_format='rfc822'),
23 'date_modified': fields.DateTime(dt_format='rfc822'),
24}This can be added at the top of the /api/controllers/files.py module or in a separate /api/utils/serializers.py module.
The difference between both serializers is that the file serializer includes the objects array in the response. We use the file_array_serializer for list responses while we use the file_serializer for object responses.
We have also made use of a function called is_allowed() to help ensure that we support all the files that we are uploading. We created a list called ALLOWED_EXTENSIONS to contain the list of all the allowed extensions.
1ALLOWED_EXTENSIONS = set(['txt', 'pdf', 'png', 'jpg', 'jpeg', 'gif'])
2
3def is_allowed(filename):
4 return '.' in filename and \
5 filename.rsplit('.', 1)[1] in ALLOWED_EXTENSIONSFinally, we conclude by adding in the resource class for ViewEditDelete in the /api/controllers/files.py module.
1class ViewEditDelete(Resource):
2 @login_required
3 @validate_user
4 @belongs_to_user
5 @marshal_with(file_serializer)
6 def get(self, user_id, file_id):
7 try:
8 should_download = request.args.get('download', False)
9 if should_download == 'true':
10 parts = os.path.split(g.file['uri'])
11 return send_from_directory(directory=parts[0], filename=parts[1])
12 return g.file
13 except Exception as e:
14 abort(500, message="There was an while processing your request --> {}".format(e.message))
15
16 @login_required
17 @validate_user
18 @belongs_to_user
19 @marshal_with(file_serializer)
20 def put(self, user_id, file_id):
21 try:
22 update_fields = {}
23 parser = reqparse.RequestParser()
24
25 parser.add_argument('name', type=str, help="New name for the file/folder")
26 parser.add_argument('parent_id', type=str, help="New parent folder for the file/folder")
27
28 args = parser.parse_args()
29
30 name = args.get('name', None)
31 parent_id = args.get('parent_id', None)
32
33 if name is not None:
34 update_fields['name'] = name
35
36 if parent_id is not None and g.file['parent_id'] != parent_id:
37 if parent_id != '0'
38 folder_access = Folder.filter({'id': parent_id, 'creator': user_id})
39 if not folder_access:
40 abort(404, message="You don't have access to the folder you're trying to move this object to")
41
42 if g.file['is_folder']:
43 update_fields['tag'] = g.file['id'] if parent_id == '0' else '{}#{}'.format(folder_access['tag'], folder['last_index'])
44 Folder.move(g.file, folder_access)
45 else:
46 File.move(g.file, folder_access)
47
48 update_fields['parent_id'] = parent_id
49
50 if g.file['is_folder']:
51 Folder.update(file_id, update_fields)
52 else:
53 File.update(file_id, update_fields)
54
55 return File.find(file_id)
56 except Exception as e:
57 abort(500, message="There was an while processing your request --> {}".format(e.message))
58
59 @login_required
60 @validate_user
61 @belongs_to_user
62 def delete(self, user_id, file_id):
63 try:
64 hard_delete = request.args.get('hard_delete', False)
65 if not g.file['is_folder']:
66 if hard_delete == 'true':
67 os.remove(g.file['uri'])
68 File.delete(file_id)
69 else:
70 File.update(file_id, {'status': False})
71 else:
72 if hard_delete == 'true':
73 folders = Folder.filter(lambda folder: folder['tag'].startswith(g.file['tag']))
74 for folder in folders:
75 files = File.filter({'parent_id': folder['id'], 'is_folder': False })
76 File.delete_where({'parent_id': folder['id'], 'is_folder': False })
77 for f in files:
78 os.remove(f['uri'])
79 else:
80 File.update(file_id, {'status': False})
81 File.update_where({'parent_id': file_id}, {'status': False})
82 return "File has been deleted successfully", 204
83 except:
84 abort(500, message="There was an error while processing your request --> {}".format(e.message))We created a get() method which returns a single file or folder object based on the ID. For folders, it includes listing information. You can see how this is done if you look at the belongs_to_user decorator. For files, we have included a query parameter should_download to be set to true if we want to download a file.
The put() method takes care of updating file and folder information. This also includes moving files and folders. Moving files is triggered by updating the parent_id field for a file/folder. The logic for both have been covered in the move() methods for the file and folder models.
The delete() method also comes a query parameter which specifies whether or not we want to perform a hard delete. For hard delete, records are removed from the database and the files are deleted from the file system. For soft delete, we only update the file status field to false.
We have created new methods here called update_where() and delete_where() in the RethinkDBModel class for deleting and updating a filtered set from the table:
1@classmethod
2def update_where(cls, predicate, fields):
3 status = r.table(cls._table).filter(predicate).update(fields).run(conn)
4 if status['errors']:
5 raise DatabaseProcessError("Could not complete the update action")
6 return True
7
8@classmethod
9def delete_where(cls, predicate):
10 status = r.table(cls._table).filter(predicate).delete().run(conn)
11 if status['errors']:
12 raise DatabaseProcessError("Could not complete the delete action")
13 return TrueConclusion
And that's it! We're done with our file storage API. Run the API to see it in action.
You can check out the codebase for the project here. Show some love by giving this guide a thumbs up at the top right.