diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 0000000..3ece078 --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1,2 @@ +github: etesync +custom: https://www.etesync.com/contribute/#donate diff --git a/.gitignore b/.gitignore index 7f220af..2c3f6a9 100644 --- a/.gitignore +++ b/.gitignore @@ -7,9 +7,10 @@ Session.vim /.coverage /tmp /media +/.idea __pycache__ .*.swp - /etebase_server_settings.py +/secret.txt diff --git a/ChangeLog.md b/ChangeLog.md index d3adef0..a74a8af 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,5 +1,23 @@ # Changelog +## Version 0.7.0 +* Chunks: improve the chunk download endpoint to use sendfile extensions +* Chunks: support not passing chunk content if exists +* Chunks: fix chunk uploading media type to accept everything +* Gracefull handle uploading the same revision +* Pass generic context to callbacks instead of the whole view +* Fix handling of some validation errors + +## Version 0.6.1 +* Collection: save the UID on the model to use the db for enforcing uniqueness + +## Version 0.6.0 +* Fix stoken calculation performance - was VERY slow in some rare cases +* Fix issues with host verification failing with a custom port - part 2 + +## Version 0.5.3 +* Add missing migration + ## Version 0.5.2 * Fix issues with host verification failing with a custom port * Add env variable to change configuration file path. diff --git a/README.md b/README.md index dc05b31..1d39872 100644 --- a/README.md +++ b/README.md @@ -9,13 +9,17 @@ An [Etebase](https://www.etebase.com) (EteSync 2.0) server so you can run your o # Installation +## Requirements + +Etebase requires Python 3.7 or newer and has a few Python dependencies (listed in `requirements.in/base.txt`). + ## From source Before installing the Etebase server make sure you install `virtualenv` (for **Python 3**): * Arch Linux: `pacman -S python-virtualenv` * Debian/Ubuntu: `apt-get install python3-virtualenv` -* Mac/Windows/Other Linux: install virtualenv or just skip the instructions mentioning virtualenv. +* Mac/Windows (WSL)/Other Linux: install virtualenv or just skip the instructions mentioning virtualenv. Then just clone the git repo and set up this app: @@ -23,11 +27,10 @@ Then just clone the git repo and set up this app: git clone https://github.com/etesync/server.git etebase cd etebase -git checkout etebase # Set up the environment and deps -virtualenv -p python3 venv # If doesn't work, try: virtualenv3 venv -source venv/bin/activate +virtualenv -p python3 .venv # If doesn't work, try: virtualenv3 .venv +source .venv/bin/activate pip install -r requirements.txt ``` @@ -42,12 +45,14 @@ To use the easy configuration file rename it to `etebase-server.ini` and place i There is also a [wikipage](https://github.com/etesync/server/wiki/Basic-Setup-Etebase-(EteSync-v2)) detailing this basic setup. Some particular settings that should be edited are: - * [`ALLOWED_HOSTS`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-ALLOWED_HOSTS) + * [`ALLOWED_HOSTS`](https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-ALLOWED_HOSTS) -- this is the list of host/domain names or addresses on which the app -will be served - * [`DEBUG`](https://docs.djangoproject.com/en/1.11/ref/settings/#debug) +will be served. For example: `etebase.example.com` + * [`DEBUG`](https://docs.djangoproject.com/en/dev/ref/settings/#debug) -- handy for debugging, set to `False` for production - * [`SECRET_KEY`](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-SECRET_KEY) + * [`MEDIA_ROOT`](https://docs.djangoproject.com/en/dev/ref/settings/#media-root) + -- the path to the directory that will hold user data. + * [`SECRET_KEY`](https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-SECRET_KEY) -- an ephemeral secret used for various cryptographic signing and token generation purposes. See below for how default configuration of `SECRET_KEY` works for this project. @@ -61,22 +66,26 @@ Now you can initialise our django app. And you are done! You can now run the debug server just to see everything works as expected by running: ``` -./manage.py runserver 0.0.0.0:8000 +uvicorn etebase_server.asgi:application --host 0.0.0.0 --port 8000 ``` Using the debug server in production is not recommended, so please read the following section for a proper deployment. # Production deployment -There are more details about a proper production setup using Daphne and Nginx in the [wiki](https://github.com/etesync/server/wiki/Production-setup-using-Daphne-and-Nginx). - -Etebase is based on Django so you should refer to one of the following - * The instructions of the Django project [here](https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/). - * Instructions from uwsgi [here](http://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html). +There are more details about a proper production setup using uvicorn and Nginx in the [wiki](https://github.com/etesync/server/wiki/Production-setup-using-Nginx). The webserver should also be configured to serve Etebase using TLS. A guide for doing so can be found in the [wiki](https://github.com/etesync/server/wiki/Setup-HTTPS-for-Etebase) as well. +The Etebase server needs to be aware of the URL it's been served as, so make sure to forward the `Host` header to the server if using a reverse proxy. For example, you would need to use the following directive in nginx: `proxy_set_header Host $host;`. + +# Data locations and backups + +The server stores user data in two different locations that need to be backed up: +1. The database - how to backup depends on which database you use. +2. The `MEDIA_ROOT` - the path where user data is stored. + # Usage Create yourself an admin user: @@ -129,6 +138,23 @@ Here are the update steps: 4. Run the migration tool to migrate all of your data. 5. Add your new EteSync 2.0 accounts to all of your devices. -# Supporting Etebase +# License + +Etebase is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License version 3 as published by the Free Software Foundation. See the [LICENSE](./LICENSE) for more information. + +A quick summary can be found [on tldrlegal](https://tldrlegal.com/license/gnu-affero-general-public-license-v3-(agpl-3.0)). Though in even simpler terms (not part of the license, and not legal advice): you can use it in however way you want, including self-hosting and commercial offerings as long as you release the code to any modifications you have made to the server software (clients are not affected). + +## Commercial licensing + +For commercial licensing options, contact license@etebase.com + +# Financially Supporting Etebase Please consider registering an account even if you self-host in order to support the development of Etebase, or visit the [contribution](https://www.etesync.com/contribute/) for more information on how to support the service. + +Become a financial contributor and help us sustain our community! + +## Contributors ($10 / month) + +[![ilovept](https://github.com/ilovept.png?size=40)](https://github.com/ilovept) +[![ryanleesipes](https://github.com/ryanleesipes.png?size=40)](https://github.com/ryanleesipes) diff --git a/django_etebase/__init__.py b/django_etebase/__init__.py index 426fefd..99ee8b6 100644 --- a/django_etebase/__init__.py +++ b/django_etebase/__init__.py @@ -1 +1 @@ -from .app_settings import app_settings +from .app_settings_inner import app_settings diff --git a/django_etebase/admin.py b/django_etebase/admin.py deleted file mode 100644 index 8c38f3f..0000000 --- a/django_etebase/admin.py +++ /dev/null @@ -1,3 +0,0 @@ -from django.contrib import admin - -# Register your models here. diff --git a/django_etebase/app_settings.py b/django_etebase/app_settings_inner.py similarity index 84% rename from django_etebase/app_settings.py rename to django_etebase/app_settings_inner.py index 7c93f5f..41fd910 100644 --- a/django_etebase/app_settings.py +++ b/django_etebase/app_settings_inner.py @@ -11,6 +11,8 @@ # # You should have received a copy of the GNU General Public License # along with this program. If not, see . +import typing as t + from django.utils.functional import cached_property @@ -32,22 +34,20 @@ class AppSettings: return getattr(settings, self.prefix + name, dflt) @cached_property - def API_PERMISSIONS(self): # pylint: disable=invalid-name - perms = self._setting("API_PERMISSIONS", ("rest_framework.permissions.IsAuthenticated",)) + def REDIS_URI(self) -> t.Optional[str]: # pylint: disable=invalid-name + return self._setting("REDIS_URI", None) + + @cached_property + def API_PERMISSIONS_READ(self): # pylint: disable=invalid-name + perms = self._setting("API_PERMISSIONS_READ", tuple()) ret = [] for perm in perms: ret.append(self.import_from_str(perm)) return ret @cached_property - def API_AUTHENTICATORS(self): # pylint: disable=invalid-name - perms = self._setting( - "API_AUTHENTICATORS", - ( - "rest_framework.authentication.TokenAuthentication", - "rest_framework.authentication.SessionAuthentication", - ), - ) + def API_PERMISSIONS_WRITE(self): # pylint: disable=invalid-name + perms = self._setting("API_PERMISSIONS_WRITE", tuple()) ret = [] for perm in perms: ret.append(self.import_from_str(perm)) diff --git a/django_etebase/drf_msgpack/apps.py b/django_etebase/drf_msgpack/apps.py deleted file mode 100644 index 22ea2c1..0000000 --- a/django_etebase/drf_msgpack/apps.py +++ /dev/null @@ -1,5 +0,0 @@ -from django.apps import AppConfig - - -class DrfMsgpackConfig(AppConfig): - name = "drf_msgpack" diff --git a/django_etebase/drf_msgpack/parsers.py b/django_etebase/drf_msgpack/parsers.py deleted file mode 100644 index 0504a76..0000000 --- a/django_etebase/drf_msgpack/parsers.py +++ /dev/null @@ -1,14 +0,0 @@ -import msgpack - -from rest_framework.parsers import BaseParser -from rest_framework.exceptions import ParseError - - -class MessagePackParser(BaseParser): - media_type = "application/msgpack" - - def parse(self, stream, media_type=None, parser_context=None): - try: - return msgpack.unpackb(stream.read(), raw=False) - except Exception as exc: - raise ParseError("MessagePack parse error - %s" % str(exc)) diff --git a/django_etebase/drf_msgpack/renderers.py b/django_etebase/drf_msgpack/renderers.py deleted file mode 100644 index 35a4afa..0000000 --- a/django_etebase/drf_msgpack/renderers.py +++ /dev/null @@ -1,15 +0,0 @@ -import msgpack - -from rest_framework.renderers import BaseRenderer - - -class MessagePackRenderer(BaseRenderer): - media_type = "application/msgpack" - format = "msgpack" - render_style = "binary" - charset = None - - def render(self, data, media_type=None, renderer_context=None): - if data is None: - return b"" - return msgpack.packb(data, use_bin_type=True) diff --git a/django_etebase/drf_msgpack/views.py b/django_etebase/drf_msgpack/views.py deleted file mode 100644 index 91ea44a..0000000 --- a/django_etebase/drf_msgpack/views.py +++ /dev/null @@ -1,3 +0,0 @@ -from django.shortcuts import render - -# Create your views here. diff --git a/django_etebase/exceptions.py b/django_etebase/exceptions.py deleted file mode 100644 index f3aa08a..0000000 --- a/django_etebase/exceptions.py +++ /dev/null @@ -1,9 +0,0 @@ -from rest_framework import serializers, status - - -class EtebaseValidationError(serializers.ValidationError): - def __init__(self, code, detail, status_code=status.HTTP_400_BAD_REQUEST): - super().__init__( - {"code": code, "detail": detail,} - ) - self.status_code = status_code diff --git a/django_etebase/migrations/0001_initial.py b/django_etebase/migrations/0001_initial.py index 86f0fa6..42ee022 100644 --- a/django_etebase/migrations/0001_initial.py +++ b/django_etebase/migrations/0001_initial.py @@ -33,7 +33,9 @@ class Migration(migrations.Migration): ("version", models.PositiveSmallIntegerField()), ("owner", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], - options={"unique_together": {("uid", "owner")},}, + options={ + "unique_together": {("uid", "owner")}, + }, ), migrations.CreateModel( name="CollectionItem", @@ -61,7 +63,9 @@ class Migration(migrations.Migration): ), ), ], - options={"unique_together": {("uid", "collection")},}, + options={ + "unique_together": {("uid", "collection")}, + }, ), migrations.CreateModel( name="CollectionItemChunk", @@ -122,7 +126,9 @@ class Migration(migrations.Migration): ), ), ], - options={"unique_together": {("item", "current")},}, + options={ + "unique_together": {("item", "current")}, + }, ), migrations.CreateModel( name="RevisionChunkRelation", @@ -145,7 +151,9 @@ class Migration(migrations.Migration): ), ), ], - options={"ordering": ("id",),}, + options={ + "ordering": ("id",), + }, ), migrations.CreateModel( name="CollectionMember", @@ -170,6 +178,8 @@ class Migration(migrations.Migration): ), ("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], - options={"unique_together": {("user", "collection")},}, + options={ + "unique_together": {("user", "collection")}, + }, ), ] diff --git a/django_etebase/migrations/0003_collectioninvitation.py b/django_etebase/migrations/0003_collectioninvitation.py index 1b416ab..4aa05c9 100644 --- a/django_etebase/migrations/0003_collectioninvitation.py +++ b/django_etebase/migrations/0003_collectioninvitation.py @@ -54,6 +54,8 @@ class Migration(migrations.Migration): ), ), ], - options={"unique_together": {("user", "fromMember")},}, + options={ + "unique_together": {("user", "fromMember")}, + }, ), ] diff --git a/django_etebase/migrations/0004_collectioninvitation_version.py b/django_etebase/migrations/0004_collectioninvitation_version.py index 29ae3f1..40a290e 100644 --- a/django_etebase/migrations/0004_collectioninvitation_version.py +++ b/django_etebase/migrations/0004_collectioninvitation_version.py @@ -11,6 +11,8 @@ class Migration(migrations.Migration): operations = [ migrations.AddField( - model_name="collectioninvitation", name="version", field=models.PositiveSmallIntegerField(default=1), + model_name="collectioninvitation", + name="version", + field=models.PositiveSmallIntegerField(default=1), ), ] diff --git a/django_etebase/migrations/0005_auto_20200526_1021.py b/django_etebase/migrations/0005_auto_20200526_1021.py index 3775277..9855ff1 100644 --- a/django_etebase/migrations/0005_auto_20200526_1021.py +++ b/django_etebase/migrations/0005_auto_20200526_1021.py @@ -10,5 +10,9 @@ class Migration(migrations.Migration): ] operations = [ - migrations.RenameField(model_name="userinfo", old_name="pubkey", new_name="loginPubkey",), + migrations.RenameField( + model_name="userinfo", + old_name="pubkey", + new_name="loginPubkey", + ), ] diff --git a/django_etebase/migrations/0013_collectionmemberremoved.py b/django_etebase/migrations/0013_collectionmemberremoved.py index 4481e80..d8c57ea 100644 --- a/django_etebase/migrations/0013_collectionmemberremoved.py +++ b/django_etebase/migrations/0013_collectionmemberremoved.py @@ -33,6 +33,8 @@ class Migration(migrations.Migration): ), ("user", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)), ], - options={"unique_together": {("user", "collection")},}, + options={ + "unique_together": {("user", "collection")}, + }, ), ] diff --git a/django_etebase/migrations/0014_auto_20200602_1558.py b/django_etebase/migrations/0014_auto_20200602_1558.py index 42bed52..ec0de46 100644 --- a/django_etebase/migrations/0014_auto_20200602_1558.py +++ b/django_etebase/migrations/0014_auto_20200602_1558.py @@ -10,5 +10,9 @@ class Migration(migrations.Migration): ] operations = [ - migrations.RenameField(model_name="userinfo", old_name="encryptedSeckey", new_name="encryptedContent",), + migrations.RenameField( + model_name="userinfo", + old_name="encryptedSeckey", + new_name="encryptedContent", + ), ] diff --git a/django_etebase/migrations/0015_collectionitemrevision_salt.py b/django_etebase/migrations/0015_collectionitemrevision_salt.py index c4dc3e9..3a13baa 100644 --- a/django_etebase/migrations/0015_collectionitemrevision_salt.py +++ b/django_etebase/migrations/0015_collectionitemrevision_salt.py @@ -11,6 +11,8 @@ class Migration(migrations.Migration): operations = [ migrations.AddField( - model_name="collectionitemrevision", name="salt", field=models.BinaryField(default=b"", editable=True), + model_name="collectionitemrevision", + name="salt", + field=models.BinaryField(default=b"", editable=True), ), ] diff --git a/django_etebase/migrations/0016_auto_20200623_0820.py b/django_etebase/migrations/0016_auto_20200623_0820.py index a273b0d..14871a2 100644 --- a/django_etebase/migrations/0016_auto_20200623_0820.py +++ b/django_etebase/migrations/0016_auto_20200623_0820.py @@ -21,7 +21,16 @@ class Migration(migrations.Migration): to="django_etebase.CollectionItem", ), ), - migrations.AlterUniqueTogether(name="collection", unique_together=set(),), - migrations.RemoveField(model_name="collection", name="uid",), - migrations.RemoveField(model_name="collection", name="version",), + migrations.AlterUniqueTogether( + name="collection", + unique_together=set(), + ), + migrations.RemoveField( + model_name="collection", + name="uid", + ), + migrations.RemoveField( + model_name="collection", + name="version", + ), ] diff --git a/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py b/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py index 21d0337..1b69a5c 100644 --- a/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py +++ b/django_etebase/migrations/0020_remove_collectionitemrevision_salt.py @@ -10,5 +10,8 @@ class Migration(migrations.Migration): ] operations = [ - migrations.RemoveField(model_name="collectionitemrevision", name="salt",), + migrations.RemoveField( + model_name="collectionitemrevision", + name="salt", + ), ] diff --git a/django_etebase/migrations/0022_auto_20200804_1059.py b/django_etebase/migrations/0022_auto_20200804_1059.py index 60af33f..bc4bad9 100644 --- a/django_etebase/migrations/0022_auto_20200804_1059.py +++ b/django_etebase/migrations/0022_auto_20200804_1059.py @@ -10,5 +10,8 @@ class Migration(migrations.Migration): ] operations = [ - migrations.AlterUniqueTogether(name="collectionitemchunk", unique_together={("item", "uid")},), + migrations.AlterUniqueTogether( + name="collectionitemchunk", + unique_together={("item", "uid")}, + ), ] diff --git a/django_etebase/migrations/0025_auto_20200804_1216.py b/django_etebase/migrations/0025_auto_20200804_1216.py index 91bf4c8..995d275 100644 --- a/django_etebase/migrations/0025_auto_20200804_1216.py +++ b/django_etebase/migrations/0025_auto_20200804_1216.py @@ -18,6 +18,12 @@ class Migration(migrations.Migration): on_delete=django.db.models.deletion.CASCADE, related_name="chunks", to="django_etebase.Collection" ), ), - migrations.AlterUniqueTogether(name="collectionitemchunk", unique_together={("collection", "uid")},), - migrations.RemoveField(model_name="collectionitemchunk", name="item",), + migrations.AlterUniqueTogether( + name="collectionitemchunk", + unique_together={("collection", "uid")}, + ), + migrations.RemoveField( + model_name="collectionitemchunk", + name="item", + ), ] diff --git a/django_etebase/migrations/0026_auto_20200907_0752.py b/django_etebase/migrations/0026_auto_20200907_0752.py index 3283654..1c90610 100644 --- a/django_etebase/migrations/0026_auto_20200907_0752.py +++ b/django_etebase/migrations/0026_auto_20200907_0752.py @@ -10,6 +10,14 @@ class Migration(migrations.Migration): ] operations = [ - migrations.RenameField(model_name="collectioninvitation", old_name="accessLevel", new_name="accessLevelOld",), - migrations.RenameField(model_name="collectionmember", old_name="accessLevel", new_name="accessLevelOld",), + migrations.RenameField( + model_name="collectioninvitation", + old_name="accessLevel", + new_name="accessLevelOld", + ), + migrations.RenameField( + model_name="collectionmember", + old_name="accessLevel", + new_name="accessLevelOld", + ), ] diff --git a/django_etebase/migrations/0029_auto_20200907_0801.py b/django_etebase/migrations/0029_auto_20200907_0801.py index f3bfe61..1e00ffe 100644 --- a/django_etebase/migrations/0029_auto_20200907_0801.py +++ b/django_etebase/migrations/0029_auto_20200907_0801.py @@ -10,6 +10,12 @@ class Migration(migrations.Migration): ] operations = [ - migrations.RemoveField(model_name="collectioninvitation", name="accessLevelOld",), - migrations.RemoveField(model_name="collectionmember", name="accessLevelOld",), + migrations.RemoveField( + model_name="collectioninvitation", + name="accessLevelOld", + ), + migrations.RemoveField( + model_name="collectionmember", + name="accessLevelOld", + ), ] diff --git a/django_etebase/migrations/0032_auto_20201013_1409.py b/django_etebase/migrations/0032_auto_20201013_1409.py index 2bb3cb0..c6e92a2 100644 --- a/django_etebase/migrations/0032_auto_20201013_1409.py +++ b/django_etebase/migrations/0032_auto_20201013_1409.py @@ -13,6 +13,6 @@ class Migration(migrations.Migration): migrations.AlterField( model_name="collectiontype", name="uid", - field=models.BinaryField(db_index=True, editable=True, unique=True), + field=models.BinaryField(db_index=True, editable=True, max_length=1024, unique=True), ), ] diff --git a/django_etebase/migrations/0033_collection_uid.py b/django_etebase/migrations/0033_collection_uid.py new file mode 100644 index 0000000..e4a829e --- /dev/null +++ b/django_etebase/migrations/0033_collection_uid.py @@ -0,0 +1,19 @@ +# Generated by Django 3.1.1 on 2020-12-14 11:21 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0032_auto_20201013_1409'), + ] + + operations = [ + migrations.AddField( + model_name='collection', + name='uid', + field=models.CharField(db_index=True, max_length=43, null=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + ), + ] diff --git a/django_etebase/migrations/0034_auto_20201214_1124.py b/django_etebase/migrations/0034_auto_20201214_1124.py new file mode 100644 index 0000000..b05060b --- /dev/null +++ b/django_etebase/migrations/0034_auto_20201214_1124.py @@ -0,0 +1,22 @@ +# Generated by Django 3.1.1 on 2020-12-14 11:24 + +from django.db import migrations + + +def update_collection_uid(apps, schema_editor): + Collection = apps.get_model("django_etebase", "Collection") + + for collection in Collection.objects.all(): + collection.uid = collection.main_item.uid + collection.save() + + +class Migration(migrations.Migration): + + dependencies = [ + ("django_etebase", "0033_collection_uid"), + ] + + operations = [ + migrations.RunPython(update_collection_uid), + ] diff --git a/django_etebase/migrations/0035_auto_20201214_1126.py b/django_etebase/migrations/0035_auto_20201214_1126.py new file mode 100644 index 0000000..489c5e0 --- /dev/null +++ b/django_etebase/migrations/0035_auto_20201214_1126.py @@ -0,0 +1,19 @@ +# Generated by Django 3.1.1 on 2020-12-14 11:26 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0034_auto_20201214_1124'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='uid', + field=models.CharField(db_index=True, max_length=43, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + ), + ] diff --git a/django_etebase/migrations/0036_auto_20201214_1128.py b/django_etebase/migrations/0036_auto_20201214_1128.py new file mode 100644 index 0000000..241adf1 --- /dev/null +++ b/django_etebase/migrations/0036_auto_20201214_1128.py @@ -0,0 +1,19 @@ +# Generated by Django 3.1.1 on 2020-12-14 11:28 + +import django.core.validators +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('django_etebase', '0035_auto_20201214_1126'), + ] + + operations = [ + migrations.AlterField( + model_name='collection', + name='uid', + field=models.CharField(db_index=True, max_length=43, unique=True, validators=[django.core.validators.RegexValidator(message='Not a valid UID', regex='^[a-zA-Z0-9\\-_]{20,}$')]), + ), + ] diff --git a/django_etebase/migrations/0037_auto_20210127_1237.py b/django_etebase/migrations/0037_auto_20210127_1237.py new file mode 100644 index 0000000..deaba1e --- /dev/null +++ b/django_etebase/migrations/0037_auto_20210127_1237.py @@ -0,0 +1,18 @@ +# Generated by Django 3.1.1 on 2021-01-27 12:37 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ("django_etebase", "0036_auto_20201214_1128"), + ] + + operations = [ + migrations.AlterField( + model_name="collectiontype", + name="uid", + field=models.BinaryField(db_index=True, editable=True, max_length=1024, unique=True), + ), + ] diff --git a/django_etebase/models.py b/django_etebase/models.py index 691947d..4640e6d 100644 --- a/django_etebase/models.py +++ b/django_etebase/models.py @@ -12,76 +12,70 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . +import typing as t from pathlib import Path from django.db import models, transaction from django.conf import settings from django.core.validators import RegexValidator -from django.db.models import Q +from django.db.models import Max, Value as V +from django.db.models.functions import Coalesce, Greatest from django.utils.functional import cached_property from django.utils.crypto import get_random_string -from rest_framework import status - from . import app_settings -from .exceptions import EtebaseValidationError UidValidator = RegexValidator(regex=r"^[a-zA-Z0-9\-_]{20,}$", message="Not a valid UID") +def stoken_annotation_builder(stoken_id_fields: t.List[str]): + aggr_fields = [Coalesce(Max(field), V(0)) for field in stoken_id_fields] + return Greatest(*aggr_fields) if len(aggr_fields) > 1 else aggr_fields[0] + + class CollectionType(models.Model): owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) - uid = models.BinaryField(editable=True, blank=False, null=False, db_index=True, unique=True) + uid = models.BinaryField(editable=True, blank=False, null=False, db_index=True, unique=True, max_length=1024) + + objects: models.manager.BaseManager["CollectionType"] class Collection(models.Model): main_item = models.OneToOneField("CollectionItem", related_name="parent", null=True, on_delete=models.SET_NULL) + # The same as main_item.uid, we just also save it here so we have DB constraints for uniqueness (and efficiency) + uid = models.CharField(db_index=True, unique=True, blank=False, max_length=43, validators=[UidValidator]) owner = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) + stoken_annotation = stoken_annotation_builder(["items__revisions__stoken", "members__stoken"]) + + objects: models.manager.BaseManager["Collection"] + def __str__(self): return self.uid - @cached_property - def uid(self): - return self.main_item.uid - @property - def content(self): + def content(self) -> "CollectionItemRevision": + assert self.main_item is not None return self.main_item.content @property - def etag(self): + def etag(self) -> str: return self.content.uid @cached_property - def stoken(self): - stoken = ( - Stoken.objects.filter( - Q(collectionitemrevision__item__collection=self) | Q(collectionmember__collection=self) - ) - .order_by("id") - .last() + def stoken(self) -> str: + stoken_id = ( + self.__class__.objects.filter(main_item=self.main_item) + .annotate(max_stoken=self.stoken_annotation) + .values("max_stoken") + .first()["max_stoken"] ) - if stoken is None: + if stoken_id == 0: raise Exception("stoken is None. Should never happen") - return stoken.uid - - def validate_unique(self, exclude=None): - super().validate_unique(exclude=exclude) - if exclude is None or "main_item" in exclude: - return - - if ( - self.__class__.objects.filter(owner=self.owner, main_item__uid=self.main_item.uid) - .exclude(id=self.id) - .exists() - ): - raise EtebaseValidationError( - "unique_uid", "Collection with this uid already exists", status_code=status.HTTP_409_CONFLICT - ) + return Stoken.objects.get(id=stoken_id).uid class CollectionItem(models.Model): @@ -90,6 +84,10 @@ class CollectionItem(models.Model): version = models.PositiveSmallIntegerField() encryptionKey = models.BinaryField(editable=True, blank=False, null=True) + stoken_annotation = stoken_annotation_builder(["revisions__stoken"]) + + objects: models.manager.BaseManager["CollectionItem"] + class Meta: unique_together = ("uid", "collection") @@ -97,23 +95,23 @@ class CollectionItem(models.Model): return "{} {}".format(self.uid, self.collection.uid) @cached_property - def content(self): + def content(self) -> "CollectionItemRevision": return self.revisions.get(current=True) @property - def etag(self): + def etag(self) -> str: return self.content.uid -def chunk_directory_path(instance, filename): +def chunk_directory_path(instance: "CollectionItemChunk", filename: str) -> Path: custom_func = app_settings.CHUNK_PATH_FUNC if custom_func is not None: return custom_func(instance, filename) - col = instance.collection - user_id = col.owner.id - uid_prefix = instance.uid[:2] - uid_rest = instance.uid[2:] + col: Collection = instance.collection + user_id: int = col.owner.id + uid_prefix: str = instance.uid[:2] + uid_rest: str = instance.uid[2:] return Path("user_{}".format(user_id), col.uid, uid_prefix, uid_rest) @@ -122,6 +120,8 @@ class CollectionItemChunk(models.Model): collection = models.ForeignKey(Collection, related_name="chunks", on_delete=models.CASCADE) chunkFile = models.FileField(upload_to=chunk_directory_path, max_length=150, unique=True) + objects: models.manager.BaseManager["CollectionItemChunk"] + def __str__(self): return self.uid @@ -134,6 +134,7 @@ def generate_stoken_uid(): class Stoken(models.Model): + id: int uid = models.CharField( db_index=True, unique=True, @@ -144,6 +145,8 @@ class Stoken(models.Model): validators=[UidValidator], ) + objects: models.manager.BaseManager["Stoken"] + class CollectionItemRevision(models.Model): stoken = models.OneToOneField(Stoken, on_delete=models.PROTECT) @@ -155,6 +158,8 @@ class CollectionItemRevision(models.Model): current = models.BooleanField(db_index=True, default=True, null=True) deleted = models.BooleanField(default=False) + objects: models.manager.BaseManager["CollectionItemRevision"] + class Meta: unique_together = ("item", "current") @@ -166,6 +171,8 @@ class RevisionChunkRelation(models.Model): chunk = models.ForeignKey(CollectionItemChunk, related_name="revisions_relation", on_delete=models.CASCADE) revision = models.ForeignKey(CollectionItemRevision, related_name="chunks_relation", on_delete=models.CASCADE) + objects: models.manager.BaseManager["RevisionChunkRelation"] + class Meta: ordering = ("id",) @@ -182,7 +189,14 @@ class CollectionMember(models.Model): user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) encryptionKey = models.BinaryField(editable=True, blank=False, null=False) collectionType = models.ForeignKey(CollectionType, on_delete=models.PROTECT, null=True) - accessLevel = models.IntegerField(choices=AccessLevels.choices, default=AccessLevels.READ_ONLY,) + accessLevel = models.IntegerField( + choices=AccessLevels.choices, + default=AccessLevels.READ_ONLY, + ) + + stoken_annotation = stoken_annotation_builder(["stoken"]) + + objects: models.manager.BaseManager["CollectionMember"] class Meta: unique_together = ("user", "collection") @@ -193,7 +207,11 @@ class CollectionMember(models.Model): def revoke(self): with transaction.atomic(): CollectionMemberRemoved.objects.update_or_create( - collection=self.collection, user=self.user, defaults={"stoken": Stoken.objects.create(),}, + collection=self.collection, + user=self.user, + defaults={ + "stoken": Stoken.objects.create(), + }, ) self.delete() @@ -204,6 +222,8 @@ class CollectionMemberRemoved(models.Model): collection = models.ForeignKey(Collection, related_name="removed_members", on_delete=models.CASCADE) user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE) + objects: models.manager.BaseManager["CollectionMemberRemoved"] + class Meta: unique_together = ("user", "collection") @@ -220,7 +240,12 @@ class CollectionInvitation(models.Model): user = models.ForeignKey(settings.AUTH_USER_MODEL, related_name="incoming_invitations", on_delete=models.CASCADE) signedEncryptionKey = models.BinaryField(editable=False, blank=False, null=False) - accessLevel = models.IntegerField(choices=AccessLevels.choices, default=AccessLevels.READ_ONLY,) + accessLevel = models.IntegerField( + choices=AccessLevels.choices, + default=AccessLevels.READ_ONLY, + ) + + objects: models.manager.BaseManager["CollectionInvitation"] class Meta: unique_together = ("user", "fromMember") @@ -229,7 +254,7 @@ class CollectionInvitation(models.Model): return "{} {}".format(self.fromMember.collection.uid, self.user) @cached_property - def collection(self): + def collection(self) -> Collection: return self.fromMember.collection @@ -241,5 +266,7 @@ class UserInfo(models.Model): encryptedContent = models.BinaryField(editable=True, blank=False, null=False) salt = models.BinaryField(editable=True, blank=False, null=False) + objects: models.manager.BaseManager["UserInfo"] + def __str__(self): return "UserInfo<{}>".format(self.owner) diff --git a/django_etebase/parsers.py b/django_etebase/parsers.py deleted file mode 100644 index c7fe58c..0000000 --- a/django_etebase/parsers.py +++ /dev/null @@ -1,16 +0,0 @@ -from rest_framework.parsers import FileUploadParser - - -class ChunkUploadParser(FileUploadParser): - """ - Parser for chunk upload data. - """ - - media_type = "application/octet-stream" - - def get_filename(self, stream, media_type, parser_context): - """ - Detects the uploaded file name. - """ - view = parser_context["view"] - return parser_context["kwargs"][view.lookup_field] diff --git a/django_etebase/permissions.py b/django_etebase/permissions.py deleted file mode 100644 index 3c77d06..0000000 --- a/django_etebase/permissions.py +++ /dev/null @@ -1,93 +0,0 @@ -# Copyright © 2017 Tom Hacohen -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, version 3. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -from rest_framework import permissions -from django_etebase.models import Collection, AccessLevels - - -def is_collection_admin(collection, user): - member = collection.members.filter(user=user).first() - return (member is not None) and (member.accessLevel == AccessLevels.ADMIN) - - -class IsCollectionAdmin(permissions.BasePermission): - """ - Custom permission to only allow owners of a collection to view it - """ - - message = { - "detail": "Only collection admins can perform this operation.", - "code": "admin_access_required", - } - - def has_permission(self, request, view): - collection_uid = view.kwargs["collection_uid"] - try: - collection = view.get_collection_queryset().get(main_item__uid=collection_uid) - return is_collection_admin(collection, request.user) - except Collection.DoesNotExist: - # If the collection does not exist, we want to 404 later, not permission denied. - return True - - -class IsCollectionAdminOrReadOnly(permissions.BasePermission): - """ - Custom permission to only allow owners of a collection to edit it - """ - - message = { - "detail": "Only collection admins can edit collections.", - "code": "admin_access_required", - } - - def has_permission(self, request, view): - collection_uid = view.kwargs.get("collection_uid", None) - - # Allow creating new collections - if collection_uid is None: - return True - - try: - collection = view.get_collection_queryset().get(main_item__uid=collection_uid) - if request.method in permissions.SAFE_METHODS: - return True - - return is_collection_admin(collection, request.user) - except Collection.DoesNotExist: - # If the collection does not exist, we want to 404 later, not permission denied. - return True - - -class HasWriteAccessOrReadOnly(permissions.BasePermission): - """ - Custom permission to restrict write - """ - - message = { - "detail": "You need write access to write to this collection", - "code": "no_write_access", - } - - def has_permission(self, request, view): - collection_uid = view.kwargs["collection_uid"] - try: - collection = view.get_collection_queryset().get(main_item__uid=collection_uid) - if request.method in permissions.SAFE_METHODS: - return True - else: - member = collection.members.get(user=request.user) - return member.accessLevel != AccessLevels.READ_ONLY - except Collection.DoesNotExist: - # If the collection does not exist, we want to 404 later, not permission denied. - return True diff --git a/django_etebase/renderers.py b/django_etebase/renderers.py deleted file mode 100644 index 0d359d3..0000000 --- a/django_etebase/renderers.py +++ /dev/null @@ -1,19 +0,0 @@ -from rest_framework.utils.encoders import JSONEncoder as DRFJSONEncoder -from rest_framework.renderers import JSONRenderer as DRFJSONRenderer - -from .serializers import b64encode - - -class JSONEncoder(DRFJSONEncoder): - def default(self, obj): - if isinstance(obj, bytes) or isinstance(obj, memoryview): - return b64encode(obj) - return super().default(obj) - - -class JSONRenderer(DRFJSONRenderer): - """ - Renderer which serializes to JSON with support for our base64 - """ - - encoder_class = JSONEncoder diff --git a/django_etebase/serializers.py b/django_etebase/serializers.py deleted file mode 100644 index ef3b296..0000000 --- a/django_etebase/serializers.py +++ /dev/null @@ -1,569 +0,0 @@ -# Copyright © 2017 Tom Hacohen -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, version 3. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -import base64 - -from django.core.files.base import ContentFile -from django.core import exceptions as django_exceptions -from django.contrib.auth import get_user_model -from django.db import IntegrityError, transaction -from rest_framework import serializers, status -from . import models -from .utils import get_user_queryset, create_user - -from .exceptions import EtebaseValidationError - -User = get_user_model() - - -def process_revisions_for_item(item, revision_data): - chunks_objs = [] - chunks = revision_data.pop("chunks_relation") - - revision = models.CollectionItemRevision(**revision_data, item=item) - revision.validate_unique() # Verify there aren't any validation issues - - for chunk in chunks: - uid = chunk[0] - chunk_obj = models.CollectionItemChunk.objects.filter(uid=uid).first() - if len(chunk) > 1: - content = chunk[1] - # If the chunk already exists we assume it's fine. Otherwise, we upload it. - if chunk_obj is None: - chunk_obj = models.CollectionItemChunk(uid=uid, collection=item.collection) - chunk_obj.chunkFile.save("IGNORED", ContentFile(content)) - chunk_obj.save() - else: - if chunk_obj is None: - raise EtebaseValidationError("chunk_no_content", "Tried to create a new chunk without content") - - chunks_objs.append(chunk_obj) - - stoken = models.Stoken.objects.create() - revision.stoken = stoken - revision.save() - - for chunk in chunks_objs: - models.RevisionChunkRelation.objects.create(chunk=chunk, revision=revision) - return revision - - -def b64encode(value): - return base64.urlsafe_b64encode(value).decode("ascii").strip("=") - - -def b64decode(data): - data += "=" * ((4 - len(data) % 4) % 4) - return base64.urlsafe_b64decode(data) - - -def b64decode_or_bytes(data): - if isinstance(data, bytes): - return data - else: - return b64decode(data) - - -class BinaryBase64Field(serializers.Field): - def to_representation(self, value): - return value - - def to_internal_value(self, data): - return b64decode_or_bytes(data) - - -class CollectionEncryptionKeyField(BinaryBase64Field): - def get_attribute(self, instance): - request = self.context.get("request", None) - if request is not None: - return instance.members.get(user=request.user).encryptionKey - return None - - -class CollectionTypeField(BinaryBase64Field): - def get_attribute(self, instance): - request = self.context.get("request", None) - if request is not None: - collection_type = instance.members.get(user=request.user).collectionType - return collection_type and collection_type.uid - return None - - -class UserSlugRelatedField(serializers.SlugRelatedField): - def get_queryset(self): - view = self.context.get("view", None) - return get_user_queryset(super().get_queryset(), view) - - def __init__(self, **kwargs): - super().__init__(slug_field=User.USERNAME_FIELD, **kwargs) - - def to_internal_value(self, data): - return super().to_internal_value(data.lower()) - - -class ChunksField(serializers.RelatedField): - def to_representation(self, obj): - obj = obj.chunk - if self.context.get("prefetch") == "auto": - with open(obj.chunkFile.path, "rb") as f: - return (obj.uid, f.read()) - else: - return (obj.uid,) - - def to_internal_value(self, data): - if data[0] is None or data[1] is None: - raise EtebaseValidationError("no_null", "null is not allowed") - return (data[0], b64decode_or_bytes(data[1])) - - -class BetterErrorsMixin: - @property - def errors(self): - nice = [] - errors = super().errors - for error_type in errors: - if error_type == "non_field_errors": - nice.extend(self.flatten_errors(None, errors[error_type])) - else: - nice.extend(self.flatten_errors(error_type, errors[error_type])) - if nice: - return {"code": "field_errors", "detail": "Field validations failed.", "errors": nice} - return {} - - def flatten_errors(self, field_name, errors): - ret = [] - if isinstance(errors, dict): - for error_key in errors: - error = errors[error_key] - ret.extend(self.flatten_errors("{}.{}".format(field_name, error_key), error)) - else: - for error in errors: - if hasattr(error, "detail"): - message = error.detail[0] - elif hasattr(error, "message"): - message = error.message - else: - message = str(error) - ret.append( - {"field": field_name, "code": error.code, "detail": message,} - ) - return ret - - def transform_validation_error(self, prefix, err): - if hasattr(err, "error_dict"): - errors = self.flatten_errors(prefix, err.error_dict) - elif not hasattr(err, "message"): - errors = self.flatten_errors(prefix, err.error_list) - else: - raise EtebaseValidationError(err.code, err.message) - - raise serializers.ValidationError( - {"code": "field_errors", "detail": "Field validations failed.", "errors": errors,} - ) - - -class CollectionItemChunkSerializer(BetterErrorsMixin, serializers.ModelSerializer): - class Meta: - model = models.CollectionItemChunk - fields = ("uid", "chunkFile") - - -class CollectionItemRevisionSerializer(BetterErrorsMixin, serializers.ModelSerializer): - chunks = ChunksField( - source="chunks_relation", - queryset=models.RevisionChunkRelation.objects.all(), - style={"base_template": "input.html"}, - many=True, - ) - meta = BinaryBase64Field() - - class Meta: - model = models.CollectionItemRevision - fields = ("chunks", "meta", "uid", "deleted") - extra_kwargs = { - "uid": {"validators": []}, # We deal with it in the serializers - } - - -class CollectionItemSerializer(BetterErrorsMixin, serializers.ModelSerializer): - encryptionKey = BinaryBase64Field(required=False, default=None, allow_null=True) - etag = serializers.CharField(allow_null=True, write_only=True) - content = CollectionItemRevisionSerializer(many=False) - - class Meta: - model = models.CollectionItem - fields = ("uid", "version", "encryptionKey", "content", "etag") - - def create(self, validated_data): - """Function that's called when this serializer creates an item""" - validate_etag = self.context.get("validate_etag", False) - etag = validated_data.pop("etag") - revision_data = validated_data.pop("content") - uid = validated_data.pop("uid") - - Model = self.__class__.Meta.model - - with transaction.atomic(): - instance, created = Model.objects.get_or_create(uid=uid, defaults=validated_data) - cur_etag = instance.etag if not created else None - - # If we are trying to update an up to date item, abort early and consider it a success - if cur_etag == revision_data.get("uid"): - return instance - - if validate_etag and cur_etag != etag: - raise EtebaseValidationError( - "wrong_etag", - "Wrong etag. Expected {} got {}".format(cur_etag, etag), - status_code=status.HTTP_409_CONFLICT, - ) - - if not created: - # We don't have to use select_for_update here because the unique constraint on current guards against - # the race condition. But it's a good idea because it'll lock and wait rather than fail. - current_revision = instance.revisions.filter(current=True).select_for_update().first() - current_revision.current = None - current_revision.save() - - try: - process_revisions_for_item(instance, revision_data) - except django_exceptions.ValidationError as e: - self.transform_validation_error("content", e) - - return instance - - def update(self, instance, validated_data): - # We never update, we always update in the create method - raise NotImplementedError() - - -class CollectionItemDepSerializer(BetterErrorsMixin, serializers.ModelSerializer): - etag = serializers.CharField() - - class Meta: - model = models.CollectionItem - fields = ("uid", "etag") - - def validate(self, data): - item = self.__class__.Meta.model.objects.get(uid=data["uid"]) - etag = data["etag"] - if item.etag != etag: - raise EtebaseValidationError( - "wrong_etag", - "Wrong etag. Expected {} got {}".format(item.etag, etag), - status_code=status.HTTP_409_CONFLICT, - ) - - return data - - -class CollectionItemBulkGetSerializer(BetterErrorsMixin, serializers.ModelSerializer): - etag = serializers.CharField(required=False) - - class Meta: - model = models.CollectionItem - fields = ("uid", "etag") - - -class CollectionListMultiSerializer(BetterErrorsMixin, serializers.Serializer): - collectionTypes = serializers.ListField(child=BinaryBase64Field()) - - -class CollectionSerializer(BetterErrorsMixin, serializers.ModelSerializer): - collectionKey = CollectionEncryptionKeyField() - collectionType = CollectionTypeField() - accessLevel = serializers.SerializerMethodField("get_access_level_from_context") - stoken = serializers.CharField(read_only=True) - - item = CollectionItemSerializer(many=False, source="main_item") - - class Meta: - model = models.Collection - fields = ("item", "accessLevel", "collectionKey", "collectionType", "stoken") - - def get_access_level_from_context(self, obj): - request = self.context.get("request", None) - if request is not None: - return obj.members.get(user=request.user).accessLevel - return None - - def create(self, validated_data): - """Function that's called when this serializer creates an item""" - collection_key = validated_data.pop("collectionKey") - collection_type = validated_data.pop("collectionType") - - user = validated_data.get("owner") - main_item_data = validated_data.pop("main_item") - etag = main_item_data.pop("etag") - revision_data = main_item_data.pop("content") - - instance = self.__class__.Meta.model(**validated_data) - - with transaction.atomic(): - _ = self.__class__.Meta.model.objects.select_for_update().filter(owner=user) - if etag is not None: - raise EtebaseValidationError("bad_etag", "etag is not null") - - instance.save() - main_item = models.CollectionItem.objects.create(**main_item_data, collection=instance) - - instance.main_item = main_item - - instance.full_clean() - instance.save() - - process_revisions_for_item(main_item, revision_data) - - collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) - - models.CollectionMember( - collection=instance, - stoken=models.Stoken.objects.create(), - user=user, - accessLevel=models.AccessLevels.ADMIN, - encryptionKey=collection_key, - collectionType=collection_type_obj, - ).save() - - return instance - - def update(self, instance, validated_data): - raise NotImplementedError() - - -class CollectionMemberSerializer(BetterErrorsMixin, serializers.ModelSerializer): - username = UserSlugRelatedField(source="user", read_only=True, style={"base_template": "input.html"},) - - class Meta: - model = models.CollectionMember - fields = ("username", "accessLevel") - - def create(self, validated_data): - raise NotImplementedError() - - def update(self, instance, validated_data): - with transaction.atomic(): - # We only allow updating accessLevel - access_level = validated_data.pop("accessLevel") - if instance.accessLevel != access_level: - instance.stoken = models.Stoken.objects.create() - instance.accessLevel = access_level - instance.save() - - return instance - - -class CollectionInvitationSerializer(BetterErrorsMixin, serializers.ModelSerializer): - username = UserSlugRelatedField(source="user", queryset=User.objects, style={"base_template": "input.html"},) - collection = serializers.CharField(source="collection.uid") - fromUsername = BinaryBase64Field(source="fromMember.user.username", read_only=True) - fromPubkey = BinaryBase64Field(source="fromMember.user.userinfo.pubkey", read_only=True) - signedEncryptionKey = BinaryBase64Field() - - class Meta: - model = models.CollectionInvitation - fields = ( - "username", - "uid", - "collection", - "signedEncryptionKey", - "accessLevel", - "fromUsername", - "fromPubkey", - "version", - ) - - def validate_user(self, value): - request = self.context["request"] - - if request.user.username == value.lower(): - raise EtebaseValidationError("no_self_invite", "Inviting yourself is not allowed") - return value - - def create(self, validated_data): - request = self.context["request"] - collection = validated_data.pop("collection") - - member = collection.members.get(user=request.user) - - with transaction.atomic(): - try: - return type(self).Meta.model.objects.create(**validated_data, fromMember=member) - except IntegrityError: - raise EtebaseValidationError("invitation_exists", "Invitation already exists") - - def update(self, instance, validated_data): - with transaction.atomic(): - instance.accessLevel = validated_data.pop("accessLevel") - instance.signedEncryptionKey = validated_data.pop("signedEncryptionKey") - instance.save() - - return instance - - -class InvitationAcceptSerializer(BetterErrorsMixin, serializers.Serializer): - collectionType = BinaryBase64Field() - encryptionKey = BinaryBase64Field() - - def create(self, validated_data): - - with transaction.atomic(): - invitation = self.context["invitation"] - encryption_key = validated_data.get("encryptionKey") - collection_type = validated_data.pop("collectionType") - - user = invitation.user - collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=collection_type, owner=user) - - member = models.CollectionMember.objects.create( - collection=invitation.collection, - stoken=models.Stoken.objects.create(), - user=user, - accessLevel=invitation.accessLevel, - encryptionKey=encryption_key, - collectionType=collection_type_obj, - ) - - models.CollectionMemberRemoved.objects.filter( - user=invitation.user, collection=invitation.collection - ).delete() - - invitation.delete() - - return member - - def update(self, instance, validated_data): - raise NotImplementedError() - - -class UserSerializer(BetterErrorsMixin, serializers.ModelSerializer): - pubkey = BinaryBase64Field(source="userinfo.pubkey") - encryptedContent = BinaryBase64Field(source="userinfo.encryptedContent") - - class Meta: - model = User - fields = (User.USERNAME_FIELD, User.EMAIL_FIELD, "pubkey", "encryptedContent") - - -class UserInfoPubkeySerializer(BetterErrorsMixin, serializers.ModelSerializer): - pubkey = BinaryBase64Field() - - class Meta: - model = models.UserInfo - fields = ("pubkey",) - - -class UserSignupSerializer(BetterErrorsMixin, serializers.ModelSerializer): - class Meta: - model = User - fields = (User.USERNAME_FIELD, User.EMAIL_FIELD) - extra_kwargs = { - "username": {"validators": []}, # We specifically validate in SignupSerializer - } - - -class AuthenticationSignupSerializer(BetterErrorsMixin, serializers.Serializer): - """Used both for creating new accounts and setting up existing ones for the first time. - When setting up existing ones the email is ignored." - """ - - user = UserSignupSerializer(many=False) - salt = BinaryBase64Field() - loginPubkey = BinaryBase64Field() - pubkey = BinaryBase64Field() - encryptedContent = BinaryBase64Field() - - def create(self, validated_data): - """Function that's called when this serializer creates an item""" - user_data = validated_data.pop("user") - - with transaction.atomic(): - try: - view = self.context.get("view", None) - user_queryset = get_user_queryset(User.objects.all(), view) - instance = user_queryset.get(**{User.USERNAME_FIELD: user_data["username"].lower()}) - except User.DoesNotExist: - # Create the user and save the casing the user chose as the first name - try: - instance = create_user(**user_data, password=None, first_name=user_data["username"], view=view) - instance.clean_fields() - except EtebaseValidationError as e: - raise e - except django_exceptions.ValidationError as e: - self.transform_validation_error("user", e) - except Exception as e: - raise EtebaseValidationError("generic", str(e)) - - if hasattr(instance, "userinfo"): - raise EtebaseValidationError("user_exists", "User already exists", status_code=status.HTTP_409_CONFLICT) - - models.UserInfo.objects.create(**validated_data, owner=instance) - - return instance - - def update(self, instance, validated_data): - raise NotImplementedError() - - -class AuthenticationLoginChallengeSerializer(BetterErrorsMixin, serializers.Serializer): - username = serializers.CharField(required=True) - - def create(self, validated_data): - raise NotImplementedError() - - def update(self, instance, validated_data): - raise NotImplementedError() - - -class AuthenticationLoginSerializer(BetterErrorsMixin, serializers.Serializer): - response = BinaryBase64Field() - signature = BinaryBase64Field() - - def create(self, validated_data): - raise NotImplementedError() - - def update(self, instance, validated_data): - raise NotImplementedError() - - -class AuthenticationLoginInnerSerializer(AuthenticationLoginChallengeSerializer): - challenge = BinaryBase64Field() - host = serializers.CharField() - action = serializers.CharField() - - def create(self, validated_data): - raise NotImplementedError() - - def update(self, instance, validated_data): - raise NotImplementedError() - - -class AuthenticationChangePasswordInnerSerializer(AuthenticationLoginInnerSerializer): - loginPubkey = BinaryBase64Field() - encryptedContent = BinaryBase64Field() - - class Meta: - model = models.UserInfo - fields = ("loginPubkey", "encryptedContent") - - def create(self, validated_data): - raise NotImplementedError() - - def update(self, instance, validated_data): - with transaction.atomic(): - instance.loginPubkey = validated_data.pop("loginPubkey") - instance.encryptedContent = validated_data.pop("encryptedContent") - instance.save() - - return instance diff --git a/django_etebase/tests.py b/django_etebase/tests.py deleted file mode 100644 index 7ce503c..0000000 --- a/django_etebase/tests.py +++ /dev/null @@ -1,3 +0,0 @@ -from django.test import TestCase - -# Create your tests here. diff --git a/django_etebase/token_auth/authentication.py b/django_etebase/token_auth/authentication.py deleted file mode 100644 index 7e84956..0000000 --- a/django_etebase/token_auth/authentication.py +++ /dev/null @@ -1,46 +0,0 @@ -from django.utils import timezone -from django.utils.translation import gettext_lazy as _ - -from rest_framework import exceptions -from rest_framework.authentication import TokenAuthentication as DRFTokenAuthentication - -from .models import AuthToken, get_default_expiry - - -AUTO_REFRESH = True -MIN_REFRESH_INTERVAL = 60 - - -class TokenAuthentication(DRFTokenAuthentication): - keyword = "Token" - model = AuthToken - - def authenticate_credentials(self, key): - msg = _("Invalid token.") - model = self.get_model() - try: - token = model.objects.select_related("user").get(key=key) - except model.DoesNotExist: - raise exceptions.AuthenticationFailed(msg) - - if not token.user.is_active: - raise exceptions.AuthenticationFailed(_("User inactive or deleted.")) - - if token.expiry is not None: - if token.expiry < timezone.now(): - token.delete() - raise exceptions.AuthenticationFailed(msg) - - if AUTO_REFRESH: - self.renew_token(token) - - return (token.user, token) - - def renew_token(self, auth_token): - current_expiry = auth_token.expiry - new_expiry = get_default_expiry() - # Throttle refreshing of token to avoid db writes - delta = (new_expiry - current_expiry).total_seconds() - if delta > MIN_REFRESH_INTERVAL: - auth_token.expiry = new_expiry - auth_token.save(update_fields=("expiry",)) diff --git a/django_etebase/token_auth/models.py b/django_etebase/token_auth/models.py index ac1efff..dd5ae87 100644 --- a/django_etebase/token_auth/models.py +++ b/django_etebase/token_auth/models.py @@ -1,9 +1,9 @@ -from django.contrib.auth import get_user_model from django.db import models from django.utils import timezone from django.utils.crypto import get_random_string +from myauth.models import get_typed_user_model -User = get_user_model() +User = get_typed_user_model() def generate_key(): diff --git a/django_etebase/urls.py b/django_etebase/urls.py deleted file mode 100644 index 01797c1..0000000 --- a/django_etebase/urls.py +++ /dev/null @@ -1,30 +0,0 @@ -from django.conf import settings -from django.conf.urls import include -from django.urls import path - -from rest_framework_nested import routers - -from django_etebase import views - -router = routers.DefaultRouter() -router.register(r"collection", views.CollectionViewSet) -router.register(r"authentication", views.AuthenticationViewSet, basename="authentication") -router.register(r"invitation/incoming", views.InvitationIncomingViewSet, basename="invitation_incoming") -router.register(r"invitation/outgoing", views.InvitationOutgoingViewSet, basename="invitation_outgoing") - -collections_router = routers.NestedSimpleRouter(router, r"collection", lookup="collection") -collections_router.register(r"item", views.CollectionItemViewSet, basename="collection_item") -collections_router.register(r"member", views.CollectionMemberViewSet, basename="collection_member") - -item_router = routers.NestedSimpleRouter(collections_router, r"item", lookup="collection_item") -item_router.register(r"chunk", views.CollectionItemChunkViewSet, basename="collection_items_chunk") - -if settings.DEBUG: - router.register(r"test/authentication", views.TestAuthenticationViewSet, basename="test_authentication") - -app_name = "django_etebase" -urlpatterns = [ - path("v1/", include(router.urls)), - path("v1/", include(collections_router.urls)), - path("v1/", include(item_router.urls)), -] diff --git a/django_etebase/utils.py b/django_etebase/utils.py index e496a77..3a05fd4 100644 --- a/django_etebase/utils.py +++ b/django_etebase/utils.py @@ -1,24 +1,35 @@ -from django.contrib.auth import get_user_model +import typing as t +from dataclasses import dataclass + +from django.db.models import QuerySet from django.core.exceptions import PermissionDenied +from myauth.models import UserType, get_typed_user_model from . import app_settings -User = get_user_model() +User = get_typed_user_model() -def get_user_queryset(queryset, view): +@dataclass +class CallbackContext: + """Class for passing extra context to callbacks""" + + url_kwargs: t.Dict[str, t.Any] + user: t.Optional[UserType] = None + + +def get_user_queryset(queryset: QuerySet[UserType], context: CallbackContext) -> QuerySet[UserType]: custom_func = app_settings.GET_USER_QUERYSET_FUNC if custom_func is not None: - return custom_func(queryset, view) + return custom_func(queryset, context) return queryset -def create_user(*args, **kwargs): +def create_user(context: CallbackContext, *args, **kwargs) -> UserType: custom_func = app_settings.CREATE_USER_FUNC if custom_func is not None: - return custom_func(*args, **kwargs) - _ = kwargs.pop("view") + return custom_func(context, *args, **kwargs) return User.objects.create_user(*args, **kwargs) diff --git a/django_etebase/views.py b/django_etebase/views.py deleted file mode 100644 index 7dd7526..0000000 --- a/django_etebase/views.py +++ /dev/null @@ -1,868 +0,0 @@ -# Copyright © 2017 Tom Hacohen -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, version 3. -# -# This library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -import msgpack - -from django.conf import settings -from django.contrib.auth import get_user_model, user_logged_in, user_logged_out -from django.core.exceptions import PermissionDenied -from django.db import transaction, IntegrityError -from django.db.models import Max, Value as V, Q -from django.db.models.functions import Coalesce, Greatest -from django.http import HttpResponseBadRequest, HttpResponse, Http404 -from django.shortcuts import get_object_or_404 - -from rest_framework import status -from rest_framework import viewsets -from rest_framework.decorators import action as action_decorator -from rest_framework.response import Response -from rest_framework.parsers import JSONParser, FormParser, MultiPartParser -from rest_framework.renderers import BrowsableAPIRenderer -from rest_framework.exceptions import AuthenticationFailed -from rest_framework.permissions import IsAuthenticated - -import nacl.encoding -import nacl.signing -import nacl.secret -import nacl.hash - -from .token_auth.models import AuthToken - -from .drf_msgpack.parsers import MessagePackParser -from .drf_msgpack.renderers import MessagePackRenderer - -from . import app_settings, permissions -from .renderers import JSONRenderer -from .models import ( - Collection, - CollectionItem, - CollectionItemRevision, - CollectionMember, - CollectionMemberRemoved, - CollectionInvitation, - Stoken, - UserInfo, -) -from .serializers import ( - AuthenticationChangePasswordInnerSerializer, - AuthenticationSignupSerializer, - AuthenticationLoginChallengeSerializer, - AuthenticationLoginSerializer, - AuthenticationLoginInnerSerializer, - CollectionSerializer, - CollectionItemSerializer, - CollectionItemBulkGetSerializer, - CollectionItemDepSerializer, - CollectionItemRevisionSerializer, - CollectionItemChunkSerializer, - CollectionListMultiSerializer, - CollectionMemberSerializer, - CollectionInvitationSerializer, - InvitationAcceptSerializer, - UserInfoPubkeySerializer, - UserSerializer, -) -from .utils import get_user_queryset -from .exceptions import EtebaseValidationError -from .parsers import ChunkUploadParser -from .signals import user_signed_up - -User = get_user_model() - - -def msgpack_encode(content): - return msgpack.packb(content, use_bin_type=True) - - -def msgpack_decode(content): - return msgpack.unpackb(content, raw=False) - - -class BaseViewSet(viewsets.ModelViewSet): - authentication_classes = tuple(app_settings.API_AUTHENTICATORS) - permission_classes = tuple(app_settings.API_PERMISSIONS) - renderer_classes = [JSONRenderer, MessagePackRenderer] + ([BrowsableAPIRenderer] if settings.DEBUG else []) - parser_classes = [JSONParser, MessagePackParser, FormParser, MultiPartParser] - stoken_id_fields = None - - def get_serializer_class(self): - serializer_class = self.serializer_class - - if self.request.method == "PUT": - serializer_class = getattr(self, "serializer_update_class", serializer_class) - - return serializer_class - - def get_collection_queryset(self, queryset=Collection.objects): - user = self.request.user - return queryset.filter(members__user=user) - - def get_stoken_obj_id(self, request): - return request.GET.get("stoken", None) - - def get_stoken_obj(self, request): - stoken = self.get_stoken_obj_id(request) - - if stoken is not None: - try: - return Stoken.objects.get(uid=stoken) - except Stoken.DoesNotExist: - raise EtebaseValidationError("bad_stoken", "Invalid stoken.", status_code=status.HTTP_400_BAD_REQUEST) - - return None - - def filter_by_stoken(self, request, queryset): - stoken_rev = self.get_stoken_obj(request) - - aggr_fields = [Coalesce(Max(field), V(0)) for field in self.stoken_id_fields] - max_stoken = Greatest(*aggr_fields) if len(aggr_fields) > 1 else aggr_fields[0] - queryset = queryset.annotate(max_stoken=max_stoken).order_by("max_stoken") - - if stoken_rev is not None: - queryset = queryset.filter(max_stoken__gt=stoken_rev.id) - - return queryset, stoken_rev - - def get_queryset_stoken(self, queryset): - maxid = -1 - for row in queryset: - rowmaxid = getattr(row, "max_stoken") or -1 - maxid = max(maxid, rowmaxid) - new_stoken = (maxid >= 0) and Stoken.objects.get(id=maxid) - - return new_stoken or None - - def filter_by_stoken_and_limit(self, request, queryset): - limit = int(request.GET.get("limit", 50)) - - queryset, stoken_rev = self.filter_by_stoken(request, queryset) - - result = list(queryset[: limit + 1]) - if len(result) < limit + 1: - done = True - else: - done = False - result = result[:-1] - - new_stoken_obj = self.get_queryset_stoken(result) or stoken_rev - - return result, new_stoken_obj, done - - # Change how our list works by default - def list(self, request, collection_uid=None, *args, **kwargs): - queryset = self.get_queryset() - serializer = self.get_serializer(queryset, many=True) - - ret = { - "data": serializer.data, - "done": True, # we always return all the items, so it's always done - } - - return Response(ret) - - -class CollectionViewSet(BaseViewSet): - allowed_methods = ["GET", "POST"] - permission_classes = BaseViewSet.permission_classes + (permissions.IsCollectionAdminOrReadOnly,) - queryset = Collection.objects.all() - serializer_class = CollectionSerializer - lookup_field = "main_item__uid" - lookup_url_kwarg = "uid" - stoken_id_fields = ["items__revisions__stoken__id", "members__stoken__id"] - - def get_queryset(self, queryset=None): - if queryset is None: - queryset = type(self).queryset - return self.get_collection_queryset(queryset) - - def get_serializer_context(self): - context = super().get_serializer_context() - prefetch = self.request.query_params.get("prefetch", "auto") - context.update({"request": self.request, "prefetch": prefetch}) - return context - - def destroy(self, request, uid=None, *args, **kwargs): - # FIXME: implement - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def partial_update(self, request, uid=None, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def update(self, request, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def create(self, request, *args, **kwargs): - serializer = self.get_serializer(data=request.data) - serializer.is_valid(raise_exception=True) - serializer.save(owner=self.request.user) - - return Response({}, status=status.HTTP_201_CREATED) - - def list(self, request, *args, **kwargs): - queryset = self.get_queryset() - return self.list_common(request, queryset, *args, **kwargs) - - @action_decorator(detail=False, methods=["POST"]) - def list_multi(self, request, *args, **kwargs): - serializer = CollectionListMultiSerializer(data=request.data) - serializer.is_valid(raise_exception=True) - - collection_types = serializer.validated_data["collectionTypes"] - - queryset = self.get_queryset() - # FIXME: Remove the isnull part once we attach collection types to all objects ("collection-type-migration") - queryset = queryset.filter( - Q(members__collectionType__uid__in=collection_types) | Q(members__collectionType__isnull=True) - ) - - return self.list_common(request, queryset, *args, **kwargs) - - def list_common(self, request, queryset, *args, **kwargs): - result, new_stoken_obj, done = self.filter_by_stoken_and_limit(request, queryset) - new_stoken = new_stoken_obj and new_stoken_obj.uid - - serializer = self.get_serializer(result, many=True) - - ret = { - "data": serializer.data, - "stoken": new_stoken, - "done": done, - } - - stoken_obj = self.get_stoken_obj(request) - if stoken_obj is not None: - # FIXME: honour limit? (the limit should be combined for data and this because of stoken) - remed_qs = CollectionMemberRemoved.objects.filter(user=request.user, stoken__id__gt=stoken_obj.id) - if not ret["done"]: - # We only filter by the new_stoken if we are not done. This is because if we are done, the new stoken - # can point to the most recent collection change rather than most recent removed membership. - remed_qs = remed_qs.filter(stoken__id__lte=new_stoken_obj.id) - - remed = remed_qs.values_list("collection__main_item__uid", flat=True) - if len(remed) > 0: - ret["removedMemberships"] = [{"uid": x} for x in remed] - - return Response(ret) - - -class CollectionItemViewSet(BaseViewSet): - allowed_methods = ["GET", "POST", "PUT"] - permission_classes = BaseViewSet.permission_classes + (permissions.HasWriteAccessOrReadOnly,) - queryset = CollectionItem.objects.all() - serializer_class = CollectionItemSerializer - lookup_field = "uid" - stoken_id_fields = ["revisions__stoken__id"] - - def get_queryset(self): - collection_uid = self.kwargs["collection_uid"] - try: - collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) - except Collection.DoesNotExist: - raise Http404("Collection does not exist") - # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') - queryset = type(self).queryset.filter(collection__pk=collection.pk, revisions__current=True) - - return queryset - - def get_serializer_context(self): - context = super().get_serializer_context() - prefetch = self.request.query_params.get("prefetch", "auto") - context.update({"request": self.request, "prefetch": prefetch}) - return context - - def create(self, request, collection_uid=None, *args, **kwargs): - # We create using batch and transaction - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def destroy(self, request, collection_uid=None, uid=None, *args, **kwargs): - # We can't have destroy because we need to get data from the user (in the body) such as hmac. - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def update(self, request, collection_uid=None, uid=None, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def partial_update(self, request, collection_uid=None, uid=None, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - def list(self, request, collection_uid=None, *args, **kwargs): - queryset = self.get_queryset() - - if not self.request.query_params.get("withCollection", False): - queryset = queryset.filter(parent__isnull=True) - - result, new_stoken_obj, done = self.filter_by_stoken_and_limit(request, queryset) - new_stoken = new_stoken_obj and new_stoken_obj.uid - - serializer = self.get_serializer(result, many=True) - - ret = { - "data": serializer.data, - "stoken": new_stoken, - "done": done, - } - return Response(ret) - - @action_decorator(detail=True, methods=["GET"]) - def revision(self, request, collection_uid=None, uid=None, *args, **kwargs): - col = get_object_or_404(self.get_collection_queryset(Collection.objects), main_item__uid=collection_uid) - item = get_object_or_404(col.items, uid=uid) - - limit = int(request.GET.get("limit", 50)) - iterator = request.GET.get("iterator", None) - - queryset = item.revisions.order_by("-id") - - if iterator is not None: - iterator = get_object_or_404(queryset, uid=iterator) - queryset = queryset.filter(id__lt=iterator.id) - - result = list(queryset[: limit + 1]) - if len(result) < limit + 1: - done = True - else: - done = False - result = result[:-1] - - serializer = CollectionItemRevisionSerializer(result, context=self.get_serializer_context(), many=True) - - iterator = serializer.data[-1]["uid"] if len(result) > 0 else None - - ret = { - "data": serializer.data, - "iterator": iterator, - "done": done, - } - return Response(ret) - - # FIXME: rename to something consistent with what the clients have - maybe list_updates? - @action_decorator(detail=False, methods=["POST"]) - def fetch_updates(self, request, collection_uid=None, *args, **kwargs): - queryset = self.get_queryset() - - serializer = CollectionItemBulkGetSerializer(data=request.data, many=True) - serializer.is_valid(raise_exception=True) - # FIXME: make configurable? - item_limit = 200 - - if len(serializer.validated_data) > item_limit: - content = {"code": "too_many_items", "detail": "Request has too many items. Limit: {}".format(item_limit)} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - queryset, stoken_rev = self.filter_by_stoken(request, queryset) - - uids, etags = zip(*[(item["uid"], item.get("etag")) for item in serializer.validated_data]) - revs = CollectionItemRevision.objects.filter(uid__in=etags, current=True) - queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) - - new_stoken_obj = self.get_queryset_stoken(queryset) - new_stoken = new_stoken_obj and new_stoken_obj.uid - stoken = stoken_rev and getattr(stoken_rev, "uid", None) - new_stoken = new_stoken or stoken - - serializer = self.get_serializer(queryset, many=True) - - ret = { - "data": serializer.data, - "stoken": new_stoken, - "done": True, # we always return all the items, so it's always done - } - return Response(ret) - - @action_decorator(detail=False, methods=["POST"]) - def batch(self, request, collection_uid=None, *args, **kwargs): - return self.transaction(request, collection_uid, validate_etag=False) - - @action_decorator(detail=False, methods=["POST"]) - def transaction(self, request, collection_uid=None, validate_etag=True, *args, **kwargs): - stoken = request.GET.get("stoken", None) - with transaction.atomic(): # We need this for locking on the collection object - collection_object = get_object_or_404( - self.get_collection_queryset(Collection.objects).select_for_update(), # Lock writes on the collection - main_item__uid=collection_uid, - ) - - if stoken is not None and stoken != collection_object.stoken: - content = {"code": "stale_stoken", "detail": "Stoken is too old"} - return Response(content, status=status.HTTP_409_CONFLICT) - - items = request.data.get("items") - deps = request.data.get("deps", None) - # FIXME: It should just be one serializer - context = self.get_serializer_context() - context.update({"validate_etag": validate_etag}) - serializer = self.get_serializer_class()(data=items, context=context, many=True) - deps_serializer = CollectionItemDepSerializer(data=deps, context=context, many=True) - - ser_valid = serializer.is_valid() - deps_ser_valid = deps is None or deps_serializer.is_valid() - if ser_valid and deps_ser_valid: - items = serializer.save(collection=collection_object) - - ret = {} - return Response(ret, status=status.HTTP_200_OK) - - return Response( - {"items": serializer.errors, "deps": deps_serializer.errors if deps is not None else [],}, - status=status.HTTP_409_CONFLICT, - ) - - -class CollectionItemChunkViewSet(viewsets.ViewSet): - allowed_methods = ["GET", "PUT"] - authentication_classes = BaseViewSet.authentication_classes - permission_classes = BaseViewSet.permission_classes - renderer_classes = BaseViewSet.renderer_classes - parser_classes = (ChunkUploadParser,) - serializer_class = CollectionItemChunkSerializer - lookup_field = "uid" - - def get_serializer_class(self): - return self.serializer_class - - def get_collection_queryset(self, queryset=Collection.objects): - user = self.request.user - return queryset.filter(members__user=user) - - def update(self, request, *args, collection_uid=None, collection_item_uid=None, uid=None, **kwargs): - col = get_object_or_404(self.get_collection_queryset(), main_item__uid=collection_uid) - # IGNORED FOR NOW: col_it = get_object_or_404(col.items, uid=collection_item_uid) - - data = { - "uid": uid, - "chunkFile": request.data["file"], - } - - serializer = self.get_serializer_class()(data=data) - serializer.is_valid(raise_exception=True) - try: - serializer.save(collection=col) - except IntegrityError: - return Response( - {"code": "chunk_exists", "detail": "Chunk already exists."}, status=status.HTTP_409_CONFLICT - ) - - return Response({}, status=status.HTTP_201_CREATED) - - @action_decorator(detail=True, methods=["GET"]) - def download(self, request, collection_uid=None, collection_item_uid=None, uid=None, *args, **kwargs): - import os - from django.views.static import serve - - col = get_object_or_404(self.get_collection_queryset(), main_item__uid=collection_uid) - # IGNORED FOR NOW: col_it = get_object_or_404(col.items, uid=collection_item_uid) - chunk = get_object_or_404(col.chunks, uid=uid) - - filename = chunk.chunkFile.path - dirname = os.path.dirname(filename) - basename = os.path.basename(filename) - - # FIXME: DO NOT USE! Use django-send file or etc instead. - return serve(request, basename, dirname) - - -class CollectionMemberViewSet(BaseViewSet): - allowed_methods = ["GET", "PUT", "DELETE"] - our_base_permission_classes = BaseViewSet.permission_classes - permission_classes = our_base_permission_classes + (permissions.IsCollectionAdmin,) - queryset = CollectionMember.objects.all() - serializer_class = CollectionMemberSerializer - lookup_field = f"user__{User.USERNAME_FIELD}__iexact" - lookup_url_kwarg = "username" - stoken_id_fields = ["stoken__id"] - - # FIXME: need to make sure that there's always an admin, and maybe also don't let an owner remove adm access - # (if we want to transfer, we need to do that specifically) - - def get_queryset(self, queryset=None): - collection_uid = self.kwargs["collection_uid"] - try: - collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) - except Collection.DoesNotExist: - raise Http404("Collection does not exist") - - if queryset is None: - queryset = type(self).queryset - - return queryset.filter(collection=collection) - - # We override this method because we expect the stoken to be called iterator - def get_stoken_obj_id(self, request): - return request.GET.get("iterator", None) - - def list(self, request, collection_uid=None, *args, **kwargs): - queryset = self.get_queryset().order_by("id") - result, new_stoken_obj, done = self.filter_by_stoken_and_limit(request, queryset) - new_stoken = new_stoken_obj and new_stoken_obj.uid - serializer = self.get_serializer(result, many=True) - - ret = { - "data": serializer.data, - "iterator": new_stoken, # Here we call it an iterator, it's only stoken for collection/items - "done": done, - } - - return Response(ret) - - def create(self, request, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - # FIXME: block leaving if we are the last admins - should be deleted / assigned in this case depending if there - # are other memebers. - def perform_destroy(self, instance): - instance.revoke() - - @action_decorator(detail=False, methods=["POST"], permission_classes=our_base_permission_classes) - def leave(self, request, collection_uid=None, *args, **kwargs): - collection_uid = self.kwargs["collection_uid"] - col = get_object_or_404(self.get_collection_queryset(Collection.objects), main_item__uid=collection_uid) - - member = col.members.get(user=request.user) - self.perform_destroy(member) - - return Response({}) - - -class InvitationBaseViewSet(BaseViewSet): - queryset = CollectionInvitation.objects.all() - serializer_class = CollectionInvitationSerializer - lookup_field = "uid" - lookup_url_kwarg = "invitation_uid" - - def list(self, request, collection_uid=None, *args, **kwargs): - limit = int(request.GET.get("limit", 50)) - iterator = request.GET.get("iterator", None) - - queryset = self.get_queryset().order_by("id") - - if iterator is not None: - iterator = get_object_or_404(queryset, uid=iterator) - queryset = queryset.filter(id__gt=iterator.id) - - result = list(queryset[: limit + 1]) - if len(result) < limit + 1: - done = True - else: - done = False - result = result[:-1] - - serializer = self.get_serializer(result, many=True) - - iterator = serializer.data[-1]["uid"] if len(result) > 0 else None - - ret = { - "data": serializer.data, - "iterator": iterator, - "done": done, - } - - return Response(ret) - - -class InvitationOutgoingViewSet(InvitationBaseViewSet): - allowed_methods = ["GET", "POST", "PUT", "DELETE"] - - def get_queryset(self, queryset=None): - if queryset is None: - queryset = type(self).queryset - - return queryset.filter(fromMember__user=self.request.user) - - def create(self, request, *args, **kwargs): - serializer = self.get_serializer(data=request.data) - serializer.is_valid(raise_exception=True) - collection_uid = serializer.validated_data.get("collection", {}).get("uid") - - try: - collection = self.get_collection_queryset(Collection.objects).get(main_item__uid=collection_uid) - except Collection.DoesNotExist: - raise Http404("Collection does not exist") - - if request.user == serializer.validated_data.get("user"): - content = {"code": "self_invite", "detail": "Inviting yourself is invalid"} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - if not permissions.is_collection_admin(collection, request.user): - raise PermissionDenied( - {"code": "admin_access_required", "detail": "User is not an admin of this collection"} - ) - - serializer.save(collection=collection) - - return Response({}, status=status.HTTP_201_CREATED) - - @action_decorator(detail=False, allowed_methods=["GET"], methods=["GET"]) - def fetch_user_profile(self, request, *args, **kwargs): - username = request.GET.get("username") - kwargs = {User.USERNAME_FIELD: username.lower()} - user = get_object_or_404(get_user_queryset(User.objects.all(), self), **kwargs) - user_info = get_object_or_404(UserInfo.objects.all(), owner=user) - serializer = UserInfoPubkeySerializer(user_info) - return Response(serializer.data) - - -class InvitationIncomingViewSet(InvitationBaseViewSet): - allowed_methods = ["GET", "DELETE"] - - def get_queryset(self, queryset=None): - if queryset is None: - queryset = type(self).queryset - - return queryset.filter(user=self.request.user) - - @action_decorator(detail=True, allowed_methods=["POST"], methods=["POST"]) - def accept(self, request, invitation_uid=None, *args, **kwargs): - invitation = get_object_or_404(self.get_queryset(), uid=invitation_uid) - context = self.get_serializer_context() - context.update({"invitation": invitation}) - - serializer = InvitationAcceptSerializer(data=request.data, context=context) - serializer.is_valid(raise_exception=True) - serializer.save() - return Response(status=status.HTTP_201_CREATED) - - -class AuthenticationViewSet(viewsets.ViewSet): - allowed_methods = ["POST"] - authentication_classes = BaseViewSet.authentication_classes - renderer_classes = BaseViewSet.renderer_classes - parser_classes = BaseViewSet.parser_classes - - def get_encryption_key(self, salt): - key = nacl.hash.blake2b(settings.SECRET_KEY.encode(), encoder=nacl.encoding.RawEncoder) - return nacl.hash.blake2b( - b"", - key=key, - salt=salt[: nacl.hash.BLAKE2B_SALTBYTES], - person=b"etebase-auth", - encoder=nacl.encoding.RawEncoder, - ) - - def get_queryset(self): - return get_user_queryset(User.objects.all(), self) - - def get_serializer_context(self): - return {"request": self.request, "format": self.format_kwarg, "view": self} - - def login_response_data(self, user): - return { - "token": AuthToken.objects.create(user=user).key, - "user": UserSerializer(user).data, - } - - def list(self, request, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - @action_decorator(detail=False, methods=["POST"]) - def signup(self, request, *args, **kwargs): - serializer = AuthenticationSignupSerializer(data=request.data, context=self.get_serializer_context()) - serializer.is_valid(raise_exception=True) - user = serializer.save() - - user_signed_up.send(sender=user.__class__, request=request, user=user) - - data = self.login_response_data(user) - return Response(data, status=status.HTTP_201_CREATED) - - def get_login_user(self, username): - kwargs = {User.USERNAME_FIELD + "__iexact": username.lower()} - try: - user = self.get_queryset().get(**kwargs) - if not hasattr(user, "userinfo"): - raise AuthenticationFailed({"code": "user_not_init", "detail": "User not properly init"}) - return user - except User.DoesNotExist: - raise AuthenticationFailed({"code": "user_not_found", "detail": "User not found"}) - - def validate_login_request(self, request, validated_data, response_raw, signature, expected_action): - from datetime import datetime - - username = validated_data.get("username") - user = self.get_login_user(username) - host = validated_data["host"] - challenge = validated_data["challenge"] - action = validated_data["action"] - - salt = bytes(user.userinfo.salt) - enc_key = self.get_encryption_key(salt) - box = nacl.secret.SecretBox(enc_key) - - challenge_data = msgpack_decode(box.decrypt(challenge)) - now = int(datetime.now().timestamp()) - if action != expected_action: - content = {"code": "wrong_action", "detail": 'Expected "{}" but got something else'.format(expected_action)} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif now - challenge_data["timestamp"] > app_settings.CHALLENGE_VALID_SECONDS: - content = {"code": "challenge_expired", "detail": "Login challange has expired"} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif challenge_data["userId"] != user.id: - content = {"code": "wrong_user", "detail": "This challenge is for the wrong user"} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - elif not settings.DEBUG and host.split(":", 1)[0] != request.get_host(): - detail = 'Found wrong host name. Got: "{}" expected: "{}"'.format(host, request.get_host()) - content = {"code": "wrong_host", "detail": detail} - return Response(content, status=status.HTTP_400_BAD_REQUEST) - - verify_key = nacl.signing.VerifyKey(bytes(user.userinfo.loginPubkey), encoder=nacl.encoding.RawEncoder) - - try: - verify_key.verify(response_raw, signature) - except nacl.exceptions.BadSignatureError: - return Response( - {"code": "login_bad_signature", "detail": "Wrong password for user."}, - status=status.HTTP_401_UNAUTHORIZED, - ) - - return None - - @action_decorator(detail=False, methods=["GET"]) - def is_etebase(self, request, *args, **kwargs): - return Response({}, status=status.HTTP_200_OK) - - @action_decorator(detail=False, methods=["POST"]) - def login_challenge(self, request, *args, **kwargs): - from datetime import datetime - - serializer = AuthenticationLoginChallengeSerializer(data=request.data) - serializer.is_valid(raise_exception=True) - username = serializer.validated_data.get("username") - user = self.get_login_user(username) - - salt = bytes(user.userinfo.salt) - enc_key = self.get_encryption_key(salt) - box = nacl.secret.SecretBox(enc_key) - - challenge_data = { - "timestamp": int(datetime.now().timestamp()), - "userId": user.id, - } - challenge = box.encrypt(msgpack_encode(challenge_data), encoder=nacl.encoding.RawEncoder) - - ret = { - "salt": salt, - "challenge": challenge, - "version": user.userinfo.version, - } - return Response(ret, status=status.HTTP_200_OK) - - @action_decorator(detail=False, methods=["POST"]) - def login(self, request, *args, **kwargs): - outer_serializer = AuthenticationLoginSerializer(data=request.data) - outer_serializer.is_valid(raise_exception=True) - - response_raw = outer_serializer.validated_data["response"] - response = msgpack_decode(response_raw) - signature = outer_serializer.validated_data["signature"] - - context = {"host": request.get_host()} - serializer = AuthenticationLoginInnerSerializer(data=response, context=context) - serializer.is_valid(raise_exception=True) - - bad_login_response = self.validate_login_request( - request, serializer.validated_data, response_raw, signature, "login" - ) - if bad_login_response is not None: - return bad_login_response - - username = serializer.validated_data.get("username") - user = self.get_login_user(username) - - data = self.login_response_data(user) - - user_logged_in.send(sender=user.__class__, request=request, user=user) - - return Response(data, status=status.HTTP_200_OK) - - @action_decorator(detail=False, methods=["POST"], permission_classes=[IsAuthenticated]) - def logout(self, request, *args, **kwargs): - request.auth.delete() - user_logged_out.send(sender=request.user.__class__, request=request, user=request.user) - return Response(status=status.HTTP_204_NO_CONTENT) - - @action_decorator(detail=False, methods=["POST"], permission_classes=BaseViewSet.permission_classes) - def change_password(self, request, *args, **kwargs): - outer_serializer = AuthenticationLoginSerializer(data=request.data) - outer_serializer.is_valid(raise_exception=True) - - response_raw = outer_serializer.validated_data["response"] - response = msgpack_decode(response_raw) - signature = outer_serializer.validated_data["signature"] - - context = {"host": request.get_host()} - serializer = AuthenticationChangePasswordInnerSerializer(request.user.userinfo, data=response, context=context) - serializer.is_valid(raise_exception=True) - - bad_login_response = self.validate_login_request( - request, serializer.validated_data, response_raw, signature, "changePassword" - ) - if bad_login_response is not None: - return bad_login_response - - serializer.save() - - return Response({}, status=status.HTTP_200_OK) - - @action_decorator(detail=False, methods=["POST"], permission_classes=[IsAuthenticated]) - def dashboard_url(self, request, *args, **kwargs): - get_dashboard_url = app_settings.DASHBOARD_URL_FUNC - if get_dashboard_url is None: - raise EtebaseValidationError( - "not_supported", "This server doesn't have a user dashboard.", status_code=status.HTTP_400_BAD_REQUEST - ) - - ret = { - "url": get_dashboard_url(request, *args, **kwargs), - } - return Response(ret) - - -class TestAuthenticationViewSet(viewsets.ViewSet): - allowed_methods = ["POST"] - renderer_classes = BaseViewSet.renderer_classes - parser_classes = BaseViewSet.parser_classes - - def get_serializer_context(self): - return {"request": self.request, "format": self.format_kwarg, "view": self} - - def list(self, request, *args, **kwargs): - return Response(status=status.HTTP_405_METHOD_NOT_ALLOWED) - - @action_decorator(detail=False, methods=["POST"]) - def reset(self, request, *args, **kwargs): - # Only run when in DEBUG mode! It's only used for tests - if not settings.DEBUG: - return HttpResponseBadRequest("Only allowed in debug mode.") - - with transaction.atomic(): - user_queryset = get_user_queryset(User.objects.all(), self) - user = get_object_or_404(user_queryset, username=request.data.get("user").get("username")) - - # Only allow test users for extra safety - if not getattr(user, User.USERNAME_FIELD).startswith("test_user"): - return HttpResponseBadRequest("Endpoint not allowed for user.") - - if hasattr(user, "userinfo"): - user.userinfo.delete() - - serializer = AuthenticationSignupSerializer(data=request.data, context=self.get_serializer_context()) - serializer.is_valid(raise_exception=True) - serializer.save() - - # Delete all of the journal data for this user for a clear test env - user.collection_set.all().delete() - user.collectionmember_set.all().delete() - user.incoming_invitations.all().delete() - - # FIXME: also delete chunk files!!! - - return HttpResponse() diff --git a/etebase-server.ini.example b/etebase-server.ini.example index 2b4682a..13903b2 100644 --- a/etebase-server.ini.example +++ b/etebase-server.ini.example @@ -8,6 +8,7 @@ debug = false ;media_url = /user-media/ ;language_code = en-us ;time_zone = UTC +;redis_uri = redis://localhost:6379 [allowed_hosts] allowed_host1 = example.com diff --git a/django_etebase/drf_msgpack/__init__.py b/etebase_fastapi/__init__.py similarity index 100% rename from django_etebase/drf_msgpack/__init__.py rename to etebase_fastapi/__init__.py diff --git a/etebase_fastapi/db_hack.py b/etebase_fastapi/db_hack.py new file mode 100644 index 0000000..24d5824 --- /dev/null +++ b/etebase_fastapi/db_hack.py @@ -0,0 +1,27 @@ +""" +FIXME: this whole function is a hack around the django db limitations due to how db connections are cached and cleaned. +Essentially django assumes there's the django request dispatcher to automatically clean up after the ORM. +""" +import typing as t +from functools import wraps + +from django.db import close_old_connections, reset_queries + + +def django_db_cleanup(): + reset_queries() + close_old_connections() + + +def django_db_cleanup_decorator(func: t.Callable[..., t.Any]): + from inspect import iscoroutinefunction + + if iscoroutinefunction(func): + return func + + @wraps(func) + def wrapper(*args, **kwargs): + django_db_cleanup() + return func(*args, **kwargs) + + return wrapper diff --git a/etebase_fastapi/dependencies.py b/etebase_fastapi/dependencies.py new file mode 100644 index 0000000..520d499 --- /dev/null +++ b/etebase_fastapi/dependencies.py @@ -0,0 +1,88 @@ +import dataclasses + +from fastapi import Depends +from fastapi.security import APIKeyHeader + +from django.utils import timezone +from django.db.models import QuerySet + +from django_etebase import models +from django_etebase.token_auth.models import AuthToken, get_default_expiry +from myauth.models import UserType, get_typed_user_model +from .exceptions import AuthenticationFailed +from .utils import get_object_or_404 +from .db_hack import django_db_cleanup_decorator + + +User = get_typed_user_model() +token_scheme = APIKeyHeader(name="Authorization") +AUTO_REFRESH = True +MIN_REFRESH_INTERVAL = 60 + + +@dataclasses.dataclass(frozen=True) +class AuthData: + user: UserType + token: AuthToken + + +def __renew_token(auth_token: AuthToken): + current_expiry = auth_token.expiry + new_expiry = get_default_expiry() + # Throttle refreshing of token to avoid db writes + delta = (new_expiry - current_expiry).total_seconds() + if delta > MIN_REFRESH_INTERVAL: + auth_token.expiry = new_expiry + auth_token.save(update_fields=("expiry",)) + + +def __get_authenticated_user(api_token: str): + api_token = api_token.split()[1] + try: + token: AuthToken = AuthToken.objects.select_related("user").get(key=api_token) + except AuthToken.DoesNotExist: + raise AuthenticationFailed(detail="Invalid token.") + if not token.user.is_active: + raise AuthenticationFailed(detail="User inactive or deleted.") + + if token.expiry is not None: + if token.expiry < timezone.now(): + token.delete() + raise AuthenticationFailed(detail="Invalid token.") + + if AUTO_REFRESH: + __renew_token(token) + + return token.user, token + + +@django_db_cleanup_decorator +def get_auth_data(api_token: str = Depends(token_scheme)) -> AuthData: + user, token = __get_authenticated_user(api_token) + return AuthData(user, token) + + +@django_db_cleanup_decorator +def get_authenticated_user(api_token: str = Depends(token_scheme)) -> UserType: + user, _ = __get_authenticated_user(api_token) + return user + + +@django_db_cleanup_decorator +def get_collection_queryset(user: UserType = Depends(get_authenticated_user)) -> QuerySet: + default_queryset: QuerySet = models.Collection.objects.all() + return default_queryset.filter(members__user=user) + + +@django_db_cleanup_decorator +def get_collection(collection_uid: str, queryset: QuerySet = Depends(get_collection_queryset)) -> models.Collection: + return get_object_or_404(queryset, uid=collection_uid) + + +@django_db_cleanup_decorator +def get_item_queryset(collection: models.Collection = Depends(get_collection)) -> QuerySet: + default_item_queryset: QuerySet = models.CollectionItem.objects.all() + # XXX Potentially add this for performance: .prefetch_related('revisions__chunks') + queryset = default_item_queryset.filter(collection__pk=collection.pk, revisions__current=True) + + return queryset diff --git a/etebase_fastapi/exceptions.py b/etebase_fastapi/exceptions.py new file mode 100644 index 0000000..1a98fcb --- /dev/null +++ b/etebase_fastapi/exceptions.py @@ -0,0 +1,128 @@ +from fastapi import status, HTTPException +import typing as t + +from pydantic import BaseModel +from django.core.exceptions import ValidationError as DjangoValidationError + + +class HttpErrorField(BaseModel): + field: str + code: str + detail: str + + class Config: + orm_mode = True + + +class HttpErrorOut(BaseModel): + code: str + detail: str + errors: t.Optional[t.List[HttpErrorField]] + + class Config: + orm_mode = True + + +class CustomHttpException(HTTPException): + def __init__(self, code: str, detail: str, status_code: int = status.HTTP_400_BAD_REQUEST): + self.code = code + super().__init__(status_code, detail) + + @property + def as_dict(self) -> dict: + return {"code": self.code, "detail": self.detail} + + +class AuthenticationFailed(CustomHttpException): + def __init__( + self, + code="authentication_failed", + detail: str = "Incorrect authentication credentials.", + status_code: int = status.HTTP_401_UNAUTHORIZED, + ): + super().__init__(code=code, detail=detail, status_code=status_code) + + +class NotAuthenticated(CustomHttpException): + def __init__( + self, + code="not_authenticated", + detail: str = "Authentication credentials were not provided.", + status_code: int = status.HTTP_401_UNAUTHORIZED, + ): + super().__init__(code=code, detail=detail, status_code=status_code) + + +class PermissionDenied(CustomHttpException): + def __init__( + self, + code="permission_denied", + detail: str = "You do not have permission to perform this action.", + status_code: int = status.HTTP_403_FORBIDDEN, + ): + super().__init__(code=code, detail=detail, status_code=status_code) + + +class NotSupported(CustomHttpException): + def __init__( + self, + code="not_implemented", + detail: str = "This server's configuration does not support this request.", + status_code: int = status.HTTP_501_NOT_IMPLEMENTED, + ): + super().__init__(code=code, detail=detail, status_code=status_code) + + +class HttpError(CustomHttpException): + def __init__( + self, + code: str, + detail: str, + status_code: int = status.HTTP_400_BAD_REQUEST, + errors: t.Optional[t.List["HttpError"]] = None, + ): + self.errors = errors + super().__init__(code=code or "generic_error", detail=detail, status_code=status_code) + + @property + def as_dict(self) -> dict: + return HttpErrorOut(code=self.code, errors=self.errors, detail=self.detail).dict() + + +class ValidationError(HttpError): + def __init__( + self, + code: str, + detail: str, + status_code: int = status.HTTP_400_BAD_REQUEST, + errors: t.Optional[t.List["HttpError"]] = None, + field: t.Optional[str] = None, + ): + self.field = field + super().__init__(code=code, detail=detail, errors=errors, status_code=status_code) + + +def flatten_errors(field_name: str, errors) -> t.List[HttpError]: + ret: t.List[HttpError] = [] + if isinstance(errors, dict): + for error_key in errors: + error = errors[error_key] + ret.extend(flatten_errors("{}.{}".format(field_name, error_key), error)) + else: + for error in errors: + if error.messages: + message = error.messages[0] + else: + message = str(error) + ret.append(ValidationError(code=error.code or "validation_error", detail=message, field=field_name)) + return ret + + +def transform_validation_error(prefix: str, err: DjangoValidationError): + if hasattr(err, "error_dict"): + errors = flatten_errors(prefix, err.error_dict) + elif not hasattr(err, "message"): + errors = flatten_errors(prefix, err.error_list) + else: + raise HttpError(err.code or "validation_error", err.message) + raise HttpError(code="field_errors", detail="Field validations failed.", errors=errors) diff --git a/etebase_fastapi/main.py b/etebase_fastapi/main.py new file mode 100644 index 0000000..d63c01d --- /dev/null +++ b/etebase_fastapi/main.py @@ -0,0 +1,77 @@ +from django.conf import settings + +# Not at the top of the file because we first need to setup django +from fastapi import FastAPI, Request +from fastapi.middleware.cors import CORSMiddleware +from fastapi.middleware.trustedhost import TrustedHostMiddleware + +from django_etebase import app_settings + +from .exceptions import CustomHttpException +from .msgpack import MsgpackResponse +from .routers.authentication import authentication_router +from .routers.collection import collection_router, item_router +from .routers.member import member_router +from .routers.invitation import invitation_incoming_router, invitation_outgoing_router +from .routers.websocket import websocket_router + + +def create_application(prefix="", middlewares=[]): + app = FastAPI( + title="Etebase", + description="The Etebase server API documentation", + externalDocs={ + "url": "https://docs.etebase.com", + "description": "Docs about the API specifications and clients.", + } + # FIXME: version="2.5.0", + ) + VERSION = "v1" + BASE_PATH = f"{prefix}/api/{VERSION}" + COLLECTION_UID_MARKER = "{collection_uid}" + app.include_router(authentication_router, prefix=f"{BASE_PATH}/authentication", tags=["authentication"]) + app.include_router(collection_router, prefix=f"{BASE_PATH}/collection", tags=["collection"]) + app.include_router(item_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}", tags=["item"]) + app.include_router(member_router, prefix=f"{BASE_PATH}/collection/{COLLECTION_UID_MARKER}", tags=["member"]) + app.include_router( + invitation_incoming_router, prefix=f"{BASE_PATH}/invitation/incoming", tags=["incoming invitation"] + ) + app.include_router( + invitation_outgoing_router, prefix=f"{BASE_PATH}/invitation/outgoing", tags=["outgoing invitation"] + ) + app.include_router(websocket_router, prefix=f"{BASE_PATH}/ws", tags=["websocket"]) + + if settings.DEBUG: + from etebase_fastapi.routers.test_reset_view import test_reset_view_router + + app.include_router(test_reset_view_router, prefix=f"{BASE_PATH}/test/authentication") + + app.add_middleware( + CORSMiddleware, + allow_origin_regex="https?://.*", + allow_credentials=True, + allow_methods=["*"], + allow_headers=["*"], + ) + app.add_middleware(TrustedHostMiddleware, allowed_hosts=settings.ALLOWED_HOSTS) + + for middleware in middlewares: + app.add_middleware(middleware) + + @app.on_event("startup") + async def on_startup() -> None: + from .redis import redisw + + await redisw.setup() + + @app.on_event("shutdown") + async def on_shutdown(): + from .redis import redisw + + await redisw.close() + + @app.exception_handler(CustomHttpException) + async def custom_exception_handler(request: Request, exc: CustomHttpException): + return MsgpackResponse(status_code=exc.status_code, content=exc.as_dict) + + return app diff --git a/etebase_fastapi/msgpack.py b/etebase_fastapi/msgpack.py new file mode 100644 index 0000000..a671e79 --- /dev/null +++ b/etebase_fastapi/msgpack.py @@ -0,0 +1,76 @@ +import typing as t + +from fastapi.routing import APIRoute, get_request_handler +from pydantic import BaseModel +from starlette.requests import Request +from starlette.responses import Response + +from .utils import msgpack_encode, msgpack_decode +from .db_hack import django_db_cleanup_decorator + + +class MsgpackRequest(Request): + media_type = "application/msgpack" + + async def json(self) -> bytes: + if not hasattr(self, "_json"): + body = await super().body() + self._json = msgpack_decode(body) + return self._json + + +class MsgpackResponse(Response): + media_type = "application/msgpack" + + def render(self, content: t.Optional[t.Any]) -> bytes: + if content is None: + return b"" + + if isinstance(content, BaseModel): + content = content.dict() + return msgpack_encode(content) + + +class MsgpackRoute(APIRoute): + # keep track of content-type -> request classes + REQUESTS_CLASSES = {MsgpackRequest.media_type: MsgpackRequest} + # keep track of content-type -> response classes + ROUTES_HANDLERS_CLASSES = {MsgpackResponse.media_type: MsgpackResponse} + + def __init__(self, path: str, endpoint: t.Callable[..., t.Any], *args, **kwargs): + endpoint = django_db_cleanup_decorator(endpoint) + super().__init__(path, endpoint, *args, **kwargs) + + def _get_media_type_route_handler(self, media_type): + return get_request_handler( + dependant=self.dependant, + body_field=self.body_field, + status_code=self.status_code, + # use custom response class or fallback on default self.response_class + response_class=self.ROUTES_HANDLERS_CLASSES.get(media_type, self.response_class), + response_field=self.secure_cloned_response_field, + response_model_include=self.response_model_include, + response_model_exclude=self.response_model_exclude, + response_model_by_alias=self.response_model_by_alias, + response_model_exclude_unset=self.response_model_exclude_unset, + response_model_exclude_defaults=self.response_model_exclude_defaults, + response_model_exclude_none=self.response_model_exclude_none, + dependency_overrides_provider=self.dependency_overrides_provider, + ) + + def get_route_handler(self) -> t.Callable: + async def custom_route_handler(request: Request) -> Response: + + content_type = request.headers.get("Content-Type") + try: + request_cls = self.REQUESTS_CLASSES[content_type] + request = request_cls(request.scope, request.receive) + except KeyError: + # nothing registered to handle content_type, process given requests as-is + pass + + accept = request.headers.get("Accept") + route_handler = self._get_media_type_route_handler(accept) + return await route_handler(request) + + return custom_route_handler diff --git a/etebase_fastapi/redis.py b/etebase_fastapi/redis.py new file mode 100644 index 0000000..3735e36 --- /dev/null +++ b/etebase_fastapi/redis.py @@ -0,0 +1,27 @@ +import typing as t +import aioredis + +from django_etebase import app_settings + + +class RedisWrapper: + redis: aioredis.Redis + + def __init__(self, redis_uri: t.Optional[str]): + self.redis_uri = redis_uri + + async def setup(self): + if self.redis_uri is not None: + self.redis = await aioredis.create_redis_pool(self.redis_uri) + + async def close(self): + if self.redis is not None: + self.redis.close() + await self.redis.wait_closed() + + @property + def is_active(self): + return self.redis_uri is not None + + +redisw = RedisWrapper(app_settings.REDIS_URI) diff --git a/django_etebase/drf_msgpack/migrations/__init__.py b/etebase_fastapi/routers/__init__.py similarity index 100% rename from django_etebase/drf_msgpack/migrations/__init__.py rename to etebase_fastapi/routers/__init__.py diff --git a/etebase_fastapi/routers/authentication.py b/etebase_fastapi/routers/authentication.py new file mode 100644 index 0000000..7ec6bc6 --- /dev/null +++ b/etebase_fastapi/routers/authentication.py @@ -0,0 +1,263 @@ +import typing as t +from typing_extensions import Literal +from datetime import datetime + +import nacl +import nacl.encoding +import nacl.hash +import nacl.secret +import nacl.signing +from django.conf import settings +from django.contrib.auth import user_logged_out, user_logged_in +from django.core import exceptions as django_exceptions +from django.db import transaction +from django.utils.functional import cached_property +from fastapi import APIRouter, Depends, status, Request + +from django_etebase import app_settings, models +from django_etebase.token_auth.models import AuthToken +from django_etebase.models import UserInfo +from django_etebase.signals import user_signed_up +from django_etebase.utils import create_user, get_user_queryset, CallbackContext +from myauth.models import UserType, get_typed_user_model +from ..exceptions import AuthenticationFailed, transform_validation_error, HttpError +from ..msgpack import MsgpackRoute +from ..utils import BaseModel, permission_responses, msgpack_encode, msgpack_decode, get_user_username_email_kwargs +from ..dependencies import AuthData, get_auth_data, get_authenticated_user + +User = get_typed_user_model() +authentication_router = APIRouter(route_class=MsgpackRoute) + + +class LoginChallengeIn(BaseModel): + username: str + + +class LoginChallengeOut(BaseModel): + salt: bytes + challenge: bytes + version: int + + +class LoginResponse(BaseModel): + username: str + challenge: bytes + host: str + action: Literal["login", "changePassword"] + + +class UserOut(BaseModel): + username: str + email: str + pubkey: bytes + encryptedContent: bytes + + @classmethod + def from_orm(cls: t.Type["UserOut"], obj: UserType) -> "UserOut": + return cls( + username=obj.username, + email=obj.email, + pubkey=bytes(obj.userinfo.pubkey), + encryptedContent=bytes(obj.userinfo.encryptedContent), + ) + + +class LoginOut(BaseModel): + token: str + user: UserOut + + @classmethod + def from_orm(cls: t.Type["LoginOut"], obj: UserType) -> "LoginOut": + token = AuthToken.objects.create(user=obj).key + user = UserOut.from_orm(obj) + return cls(token=token, user=user) + + +class Authentication(BaseModel): + class Config: + keep_untouched = (cached_property,) + + response: bytes + signature: bytes + + +class Login(Authentication): + @cached_property + def response_data(self) -> LoginResponse: + return LoginResponse(**msgpack_decode(self.response)) + + +class ChangePasswordResponse(LoginResponse): + loginPubkey: bytes + encryptedContent: bytes + + +class ChangePassword(Authentication): + @cached_property + def response_data(self) -> ChangePasswordResponse: + return ChangePasswordResponse(**msgpack_decode(self.response)) + + +class UserSignup(BaseModel): + username: str + email: str + + +class SignupIn(BaseModel): + user: UserSignup + salt: bytes + loginPubkey: bytes + pubkey: bytes + encryptedContent: bytes + + +def get_login_user(request: Request, challenge: LoginChallengeIn) -> UserType: + username = challenge.username + + kwargs = get_user_username_email_kwargs(username) + try: + user_queryset = get_user_queryset(User.objects.all(), CallbackContext(request.path_params)) + user = user_queryset.get(**kwargs) + if not hasattr(user, "userinfo"): + raise AuthenticationFailed(code="user_not_init", detail="User not properly init") + return user + except User.DoesNotExist: + raise AuthenticationFailed(code="user_not_found", detail="User not found") + + +def get_encryption_key(salt: bytes): + key = nacl.hash.blake2b(settings.SECRET_KEY.encode(), encoder=nacl.encoding.RawEncoder) + return nacl.hash.blake2b( + b"", + key=key, + salt=salt[: nacl.hash.BLAKE2B_SALTBYTES], + person=b"etebase-auth", + encoder=nacl.encoding.RawEncoder, + ) + + +def save_changed_password(data: ChangePassword, user: UserType): + response_data = data.response_data + user_info: UserInfo = user.userinfo + user_info.loginPubkey = response_data.loginPubkey + user_info.encryptedContent = response_data.encryptedContent + user_info.save() + + +def validate_login_request( + validated_data: LoginResponse, + challenge_sent_to_user: Authentication, + user: UserType, + expected_action: str, + host_from_request: str, +): + enc_key = get_encryption_key(bytes(user.userinfo.salt)) + box = nacl.secret.SecretBox(enc_key) + challenge_data = msgpack_decode(box.decrypt(validated_data.challenge)) + now = int(datetime.now().timestamp()) + if validated_data.action != expected_action: + raise HttpError("wrong_action", f'Expected "{expected_action}" but got something else') + elif now - challenge_data["timestamp"] > app_settings.CHALLENGE_VALID_SECONDS: + raise HttpError("challenge_expired", "Login challenge has expired") + elif challenge_data["userId"] != user.id: + raise HttpError("wrong_user", "This challenge is for the wrong user") + elif not settings.DEBUG and validated_data.host.split(":", 1)[0] != host_from_request: + raise HttpError( + "wrong_host", f'Found wrong host name. Got: "{validated_data.host}" expected: "{host_from_request}"' + ) + verify_key = nacl.signing.VerifyKey(bytes(user.userinfo.loginPubkey), encoder=nacl.encoding.RawEncoder) + try: + verify_key.verify(challenge_sent_to_user.response, challenge_sent_to_user.signature) + except nacl.exceptions.BadSignatureError: + raise HttpError("login_bad_signature", "Wrong password for user.", status.HTTP_401_UNAUTHORIZED) + + +@authentication_router.get("/is_etebase/") +async def is_etebase(): + pass + + +@authentication_router.post("/login_challenge/", response_model=LoginChallengeOut) +def login_challenge(user: UserType = Depends(get_login_user)): + salt = bytes(user.userinfo.salt) + enc_key = get_encryption_key(salt) + box = nacl.secret.SecretBox(enc_key) + challenge_data = { + "timestamp": int(datetime.now().timestamp()), + "userId": user.id, + } + challenge = bytes(box.encrypt(msgpack_encode(challenge_data), encoder=nacl.encoding.RawEncoder)) + return LoginChallengeOut(salt=salt, challenge=challenge, version=user.userinfo.version) + + +@authentication_router.post("/login/", response_model=LoginOut) +def login(data: Login, request: Request): + user = get_login_user(request, LoginChallengeIn(username=data.response_data.username)) + host = request.headers.get("Host") + validate_login_request(data.response_data, data, user, "login", host) + ret = LoginOut.from_orm(user) + user_logged_in.send(sender=user.__class__, request=None, user=user) + return ret + + +@authentication_router.post("/logout/", status_code=status.HTTP_204_NO_CONTENT, responses=permission_responses) +def logout(auth_data: AuthData = Depends(get_auth_data)): + auth_data.token.delete() + user_logged_out.send(sender=auth_data.user.__class__, request=None, user=auth_data.user) + + +@authentication_router.post("/change_password/", status_code=status.HTTP_204_NO_CONTENT, responses=permission_responses) +def change_password(data: ChangePassword, request: Request, user: UserType = Depends(get_authenticated_user)): + host = request.headers.get("Host") + validate_login_request(data.response_data, data, user, "changePassword", host) + save_changed_password(data, user) + + +@authentication_router.post("/dashboard_url/", responses=permission_responses) +def dashboard_url(request: Request, user: UserType = Depends(get_authenticated_user)): + get_dashboard_url = app_settings.DASHBOARD_URL_FUNC + if get_dashboard_url is None: + raise HttpError("not_supported", "This server doesn't have a user dashboard.") + + ret = { + "url": get_dashboard_url(CallbackContext(request.path_params, user=user)), + } + return ret + + +def signup_save(data: SignupIn, request: Request) -> UserType: + user_data = data.user + with transaction.atomic(): + try: + user_queryset = get_user_queryset(User.objects.all(), CallbackContext(request.path_params)) + instance = user_queryset.get(**{User.USERNAME_FIELD: user_data.username.lower()}) + except User.DoesNotExist: + # Create the user and save the casing the user chose as the first name + try: + instance = create_user( + CallbackContext(request.path_params), + **user_data.dict(), + password=None, + first_name=user_data.username, + ) + instance.full_clean() + except HttpError as e: + raise e + except django_exceptions.ValidationError as e: + transform_validation_error("user", e) + except Exception as e: + raise HttpError("generic", str(e)) + + if hasattr(instance, "userinfo"): + raise HttpError("user_exists", "User already exists", status_code=status.HTTP_409_CONFLICT) + + models.UserInfo.objects.create(**data.dict(exclude={"user"}), owner=instance) + return instance + + +@authentication_router.post("/signup/", response_model=LoginOut, status_code=status.HTTP_201_CREATED) +def signup(data: SignupIn, request: Request): + user = signup_save(data, request) + ret = LoginOut.from_orm(user) + user_signed_up.send(sender=user.__class__, request=None, user=user) + return ret diff --git a/etebase_fastapi/routers/collection.py b/etebase_fastapi/routers/collection.py new file mode 100644 index 0000000..c8146f2 --- /dev/null +++ b/etebase_fastapi/routers/collection.py @@ -0,0 +1,631 @@ +import typing as t + +from asgiref.sync import sync_to_async +from django.core import exceptions as django_exceptions +from django.core.files.base import ContentFile +from django.db import transaction, IntegrityError +from django.db.models import Q, QuerySet +from fastapi import APIRouter, Depends, status, Request, BackgroundTasks + +from django_etebase import models +from myauth.models import UserType +from .authentication import get_authenticated_user +from .websocket import get_ticket, TicketRequest, TicketOut +from ..exceptions import HttpError, transform_validation_error, PermissionDenied, ValidationError +from ..msgpack import MsgpackRoute +from ..stoken_handler import filter_by_stoken_and_limit, filter_by_stoken, get_stoken_obj, get_queryset_stoken +from ..utils import ( + get_object_or_404, + Context, + Prefetch, + PrefetchQuery, + is_collection_admin, + msgpack_encode, + BaseModel, + permission_responses, + PERMISSIONS_READ, + PERMISSIONS_READWRITE, +) +from ..dependencies import get_collection_queryset, get_item_queryset, get_collection +from ..sendfile import sendfile +from ..redis import redisw +from ..db_hack import django_db_cleanup_decorator + +collection_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) +item_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) +CollectionQuerySet = QuerySet[models.Collection] +CollectionItemQuerySet = QuerySet[models.CollectionItem] + + +class ListMulti(BaseModel): + collectionTypes: t.List[bytes] + + +ChunkType = t.Tuple[str, t.Optional[bytes]] + + +class CollectionItemRevisionInOut(BaseModel): + uid: str + meta: bytes + deleted: bool + chunks: t.List[ChunkType] + + class Config: + orm_mode = True + + @classmethod + def from_orm_context( + cls: t.Type["CollectionItemRevisionInOut"], obj: models.CollectionItemRevision, context: Context + ) -> "CollectionItemRevisionInOut": + chunks: t.List[ChunkType] = [] + for chunk_relation in obj.chunks_relation.all(): + chunk_obj = chunk_relation.chunk + if context.prefetch == "auto": + with open(chunk_obj.chunkFile.path, "rb") as f: + chunks.append((chunk_obj.uid, f.read())) + else: + chunks.append((chunk_obj.uid, None)) + return cls(uid=obj.uid, meta=bytes(obj.meta), deleted=obj.deleted, chunks=chunks) + + +class CollectionItemCommon(BaseModel): + uid: str + version: int + encryptionKey: t.Optional[bytes] + content: CollectionItemRevisionInOut + + +class CollectionItemOut(CollectionItemCommon): + class Config: + orm_mode = True + + @classmethod + def from_orm_context( + cls: t.Type["CollectionItemOut"], obj: models.CollectionItem, context: Context + ) -> "CollectionItemOut": + return cls( + uid=obj.uid, + version=obj.version, + encryptionKey=obj.encryptionKey, + content=CollectionItemRevisionInOut.from_orm_context(obj.content, context), + ) + + +class CollectionItemIn(CollectionItemCommon): + etag: t.Optional[str] + + +class CollectionCommon(BaseModel): + # FIXME: remove optional once we finish collection-type-migration + collectionType: t.Optional[bytes] + collectionKey: bytes + + +class CollectionOut(CollectionCommon): + accessLevel: models.AccessLevels + stoken: str + item: CollectionItemOut + + @classmethod + def from_orm_context(cls: t.Type["CollectionOut"], obj: models.Collection, context: Context) -> "CollectionOut": + member: models.CollectionMember = obj.members.get(user=context.user) + collection_type = member.collectionType + assert obj.main_item is not None + ret = cls( + collectionType=collection_type and bytes(collection_type.uid), + collectionKey=bytes(member.encryptionKey), + accessLevel=member.accessLevel, + stoken=obj.stoken, + item=CollectionItemOut.from_orm_context(obj.main_item, context), + ) + return ret + + +class CollectionIn(CollectionCommon): + item: CollectionItemIn + + +class RemovedMembershipOut(BaseModel): + uid: str + + +class CollectionListResponse(BaseModel): + data: t.List[CollectionOut] + stoken: t.Optional[str] + done: bool + + removedMemberships: t.Optional[t.List[RemovedMembershipOut]] + + +class CollectionItemListResponse(BaseModel): + data: t.List[CollectionItemOut] + stoken: t.Optional[str] + done: bool + + +class CollectionItemRevisionListResponse(BaseModel): + data: t.List[CollectionItemRevisionInOut] + iterator: t.Optional[str] + done: bool + + +class CollectionItemBulkGetIn(BaseModel): + uid: str + etag: t.Optional[str] + + +class ItemDepIn(BaseModel): + uid: str + etag: str + + def validate_db(self): + item = models.CollectionItem.objects.get(uid=self.uid) + etag = self.etag + if item.etag != etag: + raise ValidationError( + "wrong_etag", + "Wrong etag. Expected {} got {}".format(item.etag, etag), + status_code=status.HTTP_409_CONFLICT, + field=self.uid, + ) + + +class ItemBatchIn(BaseModel): + items: t.List[CollectionItemIn] + deps: t.Optional[t.List[ItemDepIn]] + + def validate_db(self): + if self.deps is not None: + errors: t.List[HttpError] = [] + for dep in self.deps: + try: + dep.validate_db() + except ValidationError as e: + errors.append(e) + if len(errors) > 0: + raise ValidationError( + code="dep_failed", + detail="Dependencies failed to validate", + errors=errors, + status_code=status.HTTP_409_CONFLICT, + ) + + +async def report_items_changed(col_uid: str, stoken: str, items: t.List[CollectionItemIn]): + if not redisw.is_active: + return + + redis = redisw.redis + content = msgpack_encode(CollectionItemListResponse(data=items, stoken=stoken, done=True).dict()) + await redis.publish(f"col.{col_uid}", content) + + +def collection_list_common( + queryset: CollectionQuerySet, + user: UserType, + stoken: t.Optional[str], + limit: int, + prefetch: Prefetch, +) -> CollectionListResponse: + result, new_stoken_obj, done = filter_by_stoken_and_limit( + stoken, limit, queryset, models.Collection.stoken_annotation + ) + new_stoken = new_stoken_obj and new_stoken_obj.uid + context = Context(user, prefetch) + data: t.List[CollectionOut] = [CollectionOut.from_orm_context(item, context) for item in result] + + ret = CollectionListResponse(data=data, stoken=new_stoken, done=done) + + stoken_obj = get_stoken_obj(stoken) + if stoken_obj is not None: + # FIXME: honour limit? (the limit should be combined for data and this because of stoken) + remed_qs = models.CollectionMemberRemoved.objects.filter(user=user, stoken__id__gt=stoken_obj.id) + if not done and new_stoken_obj is not None: + # We only filter by the new_stoken if we are not done. This is because if we are done, the new stoken + # can point to the most recent collection change rather than most recent removed membership. + remed_qs = remed_qs.filter(stoken__id__lte=new_stoken_obj.id) + + remed = remed_qs.values_list("collection__uid", flat=True) + if len(remed) > 0: + ret.removedMemberships = [RemovedMembershipOut(uid=x) for x in remed] + + return ret + + +# permissions + + +@django_db_cleanup_decorator +def verify_collection_admin( + collection: models.Collection = Depends(get_collection), user: UserType = Depends(get_authenticated_user) +): + if not is_collection_admin(collection, user): + raise PermissionDenied("admin_access_required", "Only collection admins can perform this operation.") + + +@django_db_cleanup_decorator +def has_write_access( + collection: models.Collection = Depends(get_collection), user: UserType = Depends(get_authenticated_user) +): + member = collection.members.get(user=user) + if member.accessLevel == models.AccessLevels.READ_ONLY: + raise PermissionDenied("no_write_access", "You need write access to write to this collection") + + +# paths + + +@collection_router.post( + "/list_multi/", + response_model=CollectionListResponse, + response_model_exclude_unset=True, + dependencies=PERMISSIONS_READ, +) +def list_multi( + data: ListMulti, + stoken: t.Optional[str] = None, + limit: int = 50, + queryset: CollectionQuerySet = Depends(get_collection_queryset), + user: UserType = Depends(get_authenticated_user), + prefetch: Prefetch = PrefetchQuery, +): + # FIXME: Remove the isnull part once we attach collection types to all objects ("collection-type-migration") + queryset = queryset.filter( + Q(members__collectionType__uid__in=data.collectionTypes) | Q(members__collectionType__isnull=True) + ) + + return collection_list_common(queryset, user, stoken, limit, prefetch) + + +@collection_router.get("/", response_model=CollectionListResponse, dependencies=PERMISSIONS_READ) +def collection_list( + stoken: t.Optional[str] = None, + limit: int = 50, + prefetch: Prefetch = PrefetchQuery, + user: UserType = Depends(get_authenticated_user), + queryset: CollectionQuerySet = Depends(get_collection_queryset), +): + return collection_list_common(queryset, user, stoken, limit, prefetch) + + +def process_revisions_for_item(item: models.CollectionItem, revision_data: CollectionItemRevisionInOut): + chunks_objs = [] + + revision = models.CollectionItemRevision(**revision_data.dict(exclude={"chunks"}), item=item) + revision.validate_unique() # Verify there aren't any validation issues + + for chunk in revision_data.chunks: + uid = chunk[0] + chunk_obj = models.CollectionItemChunk.objects.filter(uid=uid).first() + content = chunk[1] if len(chunk) > 1 else None + # If the chunk already exists we assume it's fine. Otherwise, we upload it. + if chunk_obj is None: + if content is not None: + chunk_obj = models.CollectionItemChunk(uid=uid, collection=item.collection) + chunk_obj.chunkFile.save("IGNORED", ContentFile(content)) + chunk_obj.save() + else: + raise ValidationError("chunk_no_content", "Tried to create a new chunk without content") + + chunks_objs.append(chunk_obj) + + stoken = models.Stoken.objects.create() + revision.stoken = stoken + revision.save() + + for chunk2 in chunks_objs: + models.RevisionChunkRelation.objects.create(chunk=chunk2, revision=revision) + return revision + + +def _create(data: CollectionIn, user: UserType): + with transaction.atomic(): + if data.item.etag is not None: + raise ValidationError("bad_etag", "etag is not null") + instance = models.Collection(uid=data.item.uid, owner=user) + try: + instance.validate_unique() + except django_exceptions.ValidationError: + raise ValidationError( + "unique_uid", "Collection with this uid already exists", status_code=status.HTTP_409_CONFLICT + ) + instance.save() + + main_item = models.CollectionItem.objects.create( + uid=data.item.uid, version=data.item.version, collection=instance + ) + + instance.main_item = main_item + instance.save() + + # TODO + process_revisions_for_item(main_item, data.item.content) + + collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=data.collectionType, owner=user) + + models.CollectionMember( + collection=instance, + stoken=models.Stoken.objects.create(), + user=user, + accessLevel=models.AccessLevels.ADMIN, + encryptionKey=data.collectionKey, + collectionType=collection_type_obj, + ).save() + + +@collection_router.post("/", status_code=status.HTTP_201_CREATED, dependencies=PERMISSIONS_READWRITE) +def create(data: CollectionIn, user: UserType = Depends(get_authenticated_user)): + _create(data, user) + + +@collection_router.get("/{collection_uid}/", response_model=CollectionOut, dependencies=PERMISSIONS_READ) +def collection_get( + obj: models.Collection = Depends(get_collection), + user: UserType = Depends(get_authenticated_user), + prefetch: Prefetch = PrefetchQuery, +): + return CollectionOut.from_orm_context(obj, Context(user, prefetch)) + + +def item_create(item_model: CollectionItemIn, collection: models.Collection, validate_etag: bool): + """Function that's called when this serializer creates an item""" + etag = item_model.etag + revision_data = item_model.content + uid = item_model.uid + + Model = models.CollectionItem + + with transaction.atomic(): + instance, created = Model.objects.get_or_create( + uid=uid, collection=collection, defaults=item_model.dict(exclude={"uid", "etag", "content"}) + ) + cur_etag = instance.etag if not created else None + + # If we are trying to update an up to date item, abort early and consider it a success + if cur_etag == revision_data.uid: + return instance + + if validate_etag and cur_etag != etag: + raise ValidationError( + "wrong_etag", + "Wrong etag. Expected {} got {}".format(cur_etag, etag), + status_code=status.HTTP_409_CONFLICT, + field=uid, + ) + + if not created: + # We don't have to use select_for_update here because the unique constraint on current guards against + # the race condition. But it's a good idea because it'll lock and wait rather than fail. + current_revision = instance.revisions.filter(current=True).select_for_update().first() + assert current_revision is not None + current_revision.current = None + current_revision.save() + + try: + process_revisions_for_item(instance, revision_data) + except django_exceptions.ValidationError as e: + transform_validation_error("content", e) + + return instance + + +@item_router.get("/item/{item_uid}/", response_model=CollectionItemOut, dependencies=PERMISSIONS_READ) +def item_get( + item_uid: str, + queryset: CollectionItemQuerySet = Depends(get_item_queryset), + user: UserType = Depends(get_authenticated_user), + prefetch: Prefetch = PrefetchQuery, +): + obj = queryset.get(uid=item_uid) + return CollectionItemOut.from_orm_context(obj, Context(user, prefetch)) + + +def item_list_common( + queryset: CollectionItemQuerySet, + user: UserType, + stoken: t.Optional[str], + limit: int, + prefetch: Prefetch, +) -> CollectionItemListResponse: + result, new_stoken_obj, done = filter_by_stoken_and_limit( + stoken, limit, queryset, models.CollectionItem.stoken_annotation + ) + new_stoken = new_stoken_obj and new_stoken_obj.uid + context = Context(user, prefetch) + data: t.List[CollectionItemOut] = [CollectionItemOut.from_orm_context(item, context) for item in result] + return CollectionItemListResponse(data=data, stoken=new_stoken, done=done) + + +@item_router.get("/item/", response_model=CollectionItemListResponse, dependencies=PERMISSIONS_READ) +def item_list( + queryset: CollectionItemQuerySet = Depends(get_item_queryset), + stoken: t.Optional[str] = None, + limit: int = 50, + prefetch: Prefetch = PrefetchQuery, + withCollection: bool = False, + user: UserType = Depends(get_authenticated_user), +): + if not withCollection: + queryset = queryset.filter(parent__isnull=True) + + response = item_list_common(queryset, user, stoken, limit, prefetch) + return response + + +@item_router.post("/item/subscription-ticket/", response_model=TicketOut, dependencies=PERMISSIONS_READ) +async def item_list_subscription_ticket( + collection: models.Collection = Depends(get_collection), + user: UserType = Depends(get_authenticated_user), +): + """Get an authentication ticket that can be used with the websocket endpoint""" + return await get_ticket(TicketRequest(collection=collection.uid), user) + + +def item_bulk_common( + data: ItemBatchIn, + user: UserType, + stoken: t.Optional[str], + uid: str, + validate_etag: bool, + background_tasks: BackgroundTasks, +): + queryset = get_collection_queryset(user) + with transaction.atomic(): # We need this for locking the collection object + collection_object = queryset.select_for_update().get(uid=uid) + + if stoken and stoken != collection_object.stoken: + raise HttpError("stale_stoken", "Stoken is too old", status_code=status.HTTP_409_CONFLICT) + + data.validate_db() + + errors: t.List[HttpError] = [] + for item in data.items: + try: + item_create(item, collection_object, validate_etag) + except ValidationError as e: + errors.append(e) + + if len(errors) > 0: + raise ValidationError( + code="item_failed", + detail="Items failed to validate", + errors=errors, + status_code=status.HTTP_409_CONFLICT, + ) + + background_tasks.add_task(report_items_changed, collection_object.uid, collection_object.stoken, data.items) + + +@item_router.get( + "/item/{item_uid}/revision/", response_model=CollectionItemRevisionListResponse, dependencies=PERMISSIONS_READ +) +def item_revisions( + item_uid: str, + limit: int = 50, + iterator: t.Optional[str] = None, + prefetch: Prefetch = PrefetchQuery, + user: UserType = Depends(get_authenticated_user), + items: CollectionItemQuerySet = Depends(get_item_queryset), +): + item = get_object_or_404(items, uid=item_uid) + + queryset = item.revisions.order_by("-id") + + if iterator is not None: + iterator_obj = get_object_or_404(queryset, uid=iterator) + queryset = queryset.filter(id__lt=iterator_obj.id) + + result = list(queryset[: limit + 1]) + if len(result) < limit + 1: + done = True + else: + done = False + result = result[:-1] + + context = Context(user, prefetch) + ret_data = [CollectionItemRevisionInOut.from_orm_context(revision, context) for revision in result] + iterator = ret_data[-1].uid if len(result) > 0 else None + + return CollectionItemRevisionListResponse( + data=ret_data, + iterator=iterator, + done=done, + ) + + +@item_router.post("/item/fetch_updates/", response_model=CollectionItemListResponse, dependencies=PERMISSIONS_READ) +def fetch_updates( + data: t.List[CollectionItemBulkGetIn], + stoken: t.Optional[str] = None, + prefetch: Prefetch = PrefetchQuery, + user: UserType = Depends(get_authenticated_user), + queryset: CollectionItemQuerySet = Depends(get_item_queryset), +): + # FIXME: make configurable? + item_limit = 200 + + if len(data) > item_limit: + raise HttpError("too_many_items", "Request has too many items.", status_code=status.HTTP_400_BAD_REQUEST) + + queryset, stoken_rev = filter_by_stoken(stoken, queryset, models.CollectionItem.stoken_annotation) + + uids, etags = zip(*[(item.uid, item.etag) for item in data]) + revs = models.CollectionItemRevision.objects.filter(uid__in=etags, current=True) + queryset = queryset.filter(uid__in=uids).exclude(revisions__in=revs) + + new_stoken_obj = get_queryset_stoken(queryset) + new_stoken = new_stoken_obj and new_stoken_obj.uid + stoken_rev_uid = stoken_rev and getattr(stoken_rev, "uid", None) + new_stoken = new_stoken or stoken_rev_uid + + context = Context(user, prefetch) + return CollectionItemListResponse( + data=[CollectionItemOut.from_orm_context(item, context) for item in queryset], + stoken=new_stoken, + done=True, # we always return all the items, so it's always done + ) + + +@item_router.post("/item/transaction/", dependencies=[Depends(has_write_access), *PERMISSIONS_READWRITE]) +def item_transaction( + collection_uid: str, + data: ItemBatchIn, + background_tasks: BackgroundTasks, + stoken: t.Optional[str] = None, + user: UserType = Depends(get_authenticated_user), +): + return item_bulk_common(data, user, stoken, collection_uid, validate_etag=True, background_tasks=background_tasks) + + +@item_router.post("/item/batch/", dependencies=[Depends(has_write_access), *PERMISSIONS_READWRITE]) +def item_batch( + collection_uid: str, + data: ItemBatchIn, + background_tasks: BackgroundTasks, + stoken: t.Optional[str] = None, + user: UserType = Depends(get_authenticated_user), +): + return item_bulk_common(data, user, stoken, collection_uid, validate_etag=False, background_tasks=background_tasks) + + +# Chunks + + +@sync_to_async +def chunk_save(chunk_uid: str, collection: models.Collection, content_file: ContentFile): + chunk_obj = models.CollectionItemChunk(uid=chunk_uid, collection=collection) + chunk_obj.chunkFile.save("IGNORED", content_file) + chunk_obj.save() + return chunk_obj + + +@item_router.put( + "/item/{item_uid}/chunk/{chunk_uid}/", + dependencies=[Depends(has_write_access), *PERMISSIONS_READWRITE], + status_code=status.HTTP_201_CREATED, +) +async def chunk_update( + request: Request, + chunk_uid: str, + collection: models.Collection = Depends(get_collection), +): + # IGNORED FOR NOW: col_it = get_object_or_404(col.items, uid=collection_item_uid) + content_file = ContentFile(await request.body()) + try: + await chunk_save(chunk_uid, collection, content_file) + except IntegrityError: + raise HttpError("chunk_exists", "Chunk already exists.", status_code=status.HTTP_409_CONFLICT) + + +@item_router.get( + "/item/{item_uid}/chunk/{chunk_uid}/download/", + dependencies=PERMISSIONS_READ, +) +def chunk_download( + chunk_uid: str, + collection: models.Collection = Depends(get_collection), +): + chunk = get_object_or_404(collection.chunks, uid=chunk_uid) + + filename = chunk.chunkFile.path + return sendfile(filename) diff --git a/etebase_fastapi/routers/invitation.py b/etebase_fastapi/routers/invitation.py new file mode 100644 index 0000000..7e52978 --- /dev/null +++ b/etebase_fastapi/routers/invitation.py @@ -0,0 +1,244 @@ +import typing as t + +from django.db import transaction, IntegrityError +from django.db.models import QuerySet +from fastapi import APIRouter, Depends, status, Request + +from django_etebase import models +from django_etebase.utils import get_user_queryset, CallbackContext +from myauth.models import UserType, get_typed_user_model +from .authentication import get_authenticated_user +from ..exceptions import HttpError, PermissionDenied +from ..msgpack import MsgpackRoute +from ..utils import ( + get_object_or_404, + get_user_username_email_kwargs, + Context, + is_collection_admin, + BaseModel, + permission_responses, + PERMISSIONS_READ, + PERMISSIONS_READWRITE, +) +from ..db_hack import django_db_cleanup_decorator + +User = get_typed_user_model() +invitation_incoming_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) +invitation_outgoing_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) +InvitationQuerySet = QuerySet[models.CollectionInvitation] +default_queryset: InvitationQuerySet = models.CollectionInvitation.objects.all() + + +class UserInfoOut(BaseModel): + pubkey: bytes + + class Config: + orm_mode = True + + @classmethod + def from_orm(cls: t.Type["UserInfoOut"], obj: models.UserInfo) -> "UserInfoOut": + return cls(pubkey=bytes(obj.pubkey)) + + +class CollectionInvitationAcceptIn(BaseModel): + collectionType: bytes + encryptionKey: bytes + + +class CollectionInvitationCommon(BaseModel): + uid: str + version: int + accessLevel: models.AccessLevels + username: str + collection: str + signedEncryptionKey: bytes + + +class CollectionInvitationIn(CollectionInvitationCommon): + def validate_db(self, context: Context): + user = context.user + if user is not None and (user.username == self.username.lower()): + raise HttpError("no_self_invite", "Inviting yourself is not allowed") + + +class CollectionInvitationOut(CollectionInvitationCommon): + fromUsername: str + fromPubkey: bytes + + class Config: + orm_mode = True + + @classmethod + def from_orm(cls: t.Type["CollectionInvitationOut"], obj: models.CollectionInvitation) -> "CollectionInvitationOut": + return cls( + uid=obj.uid, + version=obj.version, + accessLevel=obj.accessLevel, + username=obj.user.username, + collection=obj.collection.uid, + fromUsername=obj.fromMember.user.username, + fromPubkey=bytes(obj.fromMember.user.userinfo.pubkey), + signedEncryptionKey=bytes(obj.signedEncryptionKey), + ) + + +class InvitationListResponse(BaseModel): + data: t.List[CollectionInvitationOut] + iterator: t.Optional[str] + done: bool + + +@django_db_cleanup_decorator +def get_incoming_queryset(user: UserType = Depends(get_authenticated_user)): + return default_queryset.filter(user=user) + + +@django_db_cleanup_decorator +def get_outgoing_queryset(user: UserType = Depends(get_authenticated_user)): + return default_queryset.filter(fromMember__user=user) + + +def list_common( + queryset: InvitationQuerySet, + iterator: t.Optional[str], + limit: int, +) -> InvitationListResponse: + queryset = queryset.order_by("id") + + if iterator is not None: + iterator_obj = get_object_or_404(queryset, uid=iterator) + queryset = queryset.filter(id__gt=iterator_obj.id) + + result = list(queryset[: limit + 1]) + if len(result) < limit + 1: + done = True + else: + done = False + result = result[:-1] + + ret_data = result + iterator = ret_data[-1].uid if len(result) > 0 else None + + return InvitationListResponse( + data=ret_data, + iterator=iterator, + done=done, + ) + + +@invitation_incoming_router.get("/", response_model=InvitationListResponse, dependencies=PERMISSIONS_READ) +def incoming_list( + iterator: t.Optional[str] = None, + limit: int = 50, + queryset: InvitationQuerySet = Depends(get_incoming_queryset), +): + return list_common(queryset, iterator, limit) + + +@invitation_incoming_router.get( + "/{invitation_uid}/", response_model=CollectionInvitationOut, dependencies=PERMISSIONS_READ +) +def incoming_get( + invitation_uid: str, + queryset: InvitationQuerySet = Depends(get_incoming_queryset), +): + obj = get_object_or_404(queryset, uid=invitation_uid) + return CollectionInvitationOut.from_orm(obj) + + +@invitation_incoming_router.delete( + "/{invitation_uid}/", status_code=status.HTTP_204_NO_CONTENT, dependencies=PERMISSIONS_READWRITE +) +def incoming_delete( + invitation_uid: str, + queryset: InvitationQuerySet = Depends(get_incoming_queryset), +): + obj = get_object_or_404(queryset, uid=invitation_uid) + obj.delete() + + +@invitation_incoming_router.post( + "/{invitation_uid}/accept/", status_code=status.HTTP_201_CREATED, dependencies=PERMISSIONS_READWRITE +) +def incoming_accept( + invitation_uid: str, + data: CollectionInvitationAcceptIn, + queryset: InvitationQuerySet = Depends(get_incoming_queryset), +): + invitation = get_object_or_404(queryset, uid=invitation_uid) + + with transaction.atomic(): + user = invitation.user + collection_type_obj, _ = models.CollectionType.objects.get_or_create(uid=data.collectionType, owner=user) + + models.CollectionMember.objects.create( + collection=invitation.collection, + stoken=models.Stoken.objects.create(), + user=user, + accessLevel=invitation.accessLevel, + encryptionKey=data.encryptionKey, + collectionType=collection_type_obj, + ) + + models.CollectionMemberRemoved.objects.filter(user=invitation.user, collection=invitation.collection).delete() + + invitation.delete() + + +@invitation_outgoing_router.post("/", status_code=status.HTTP_201_CREATED, dependencies=PERMISSIONS_READWRITE) +def outgoing_create( + data: CollectionInvitationIn, + request: Request, + user: UserType = Depends(get_authenticated_user), +): + collection = get_object_or_404(models.Collection.objects, uid=data.collection) + kwargs = get_user_username_email_kwargs(data.username) + to_user = get_object_or_404(get_user_queryset(User.objects.all(), CallbackContext(request.path_params)), **kwargs) + + context = Context(user, None) + data.validate_db(context) + + if not is_collection_admin(collection, user): + raise PermissionDenied("admin_access_required", "User is not an admin of this collection") + + member = collection.members.get(user=user) + + with transaction.atomic(): + try: + models.CollectionInvitation.objects.create( + **data.dict(exclude={"collection", "username"}), user=to_user, fromMember=member + ) + except IntegrityError: + raise HttpError("invitation_exists", "Invitation already exists") + + +@invitation_outgoing_router.get("/", response_model=InvitationListResponse, dependencies=PERMISSIONS_READ) +def outgoing_list( + iterator: t.Optional[str] = None, + limit: int = 50, + queryset: InvitationQuerySet = Depends(get_outgoing_queryset), +): + return list_common(queryset, iterator, limit) + + +@invitation_outgoing_router.delete( + "/{invitation_uid}/", status_code=status.HTTP_204_NO_CONTENT, dependencies=PERMISSIONS_READWRITE +) +def outgoing_delete( + invitation_uid: str, + queryset: InvitationQuerySet = Depends(get_outgoing_queryset), +): + obj = get_object_or_404(queryset, uid=invitation_uid) + obj.delete() + + +@invitation_outgoing_router.get("/fetch_user_profile/", response_model=UserInfoOut, dependencies=PERMISSIONS_READ) +def outgoing_fetch_user_profile( + username: str, + request: Request, + user: UserType = Depends(get_authenticated_user), +): + kwargs = get_user_username_email_kwargs(username) + user = get_object_or_404(get_user_queryset(User.objects.all(), CallbackContext(request.path_params)), **kwargs) + user_info = get_object_or_404(models.UserInfo.objects.all(), owner=user) + return UserInfoOut.from_orm(user_info) diff --git a/etebase_fastapi/routers/member.py b/etebase_fastapi/routers/member.py new file mode 100644 index 0000000..38beb79 --- /dev/null +++ b/etebase_fastapi/routers/member.py @@ -0,0 +1,109 @@ +import typing as t + +from django.db import transaction +from django.db.models import QuerySet +from fastapi import APIRouter, Depends, status + +from django_etebase import models +from myauth.models import UserType, get_typed_user_model +from .authentication import get_authenticated_user +from ..msgpack import MsgpackRoute +from ..utils import get_object_or_404, BaseModel, permission_responses, PERMISSIONS_READ, PERMISSIONS_READWRITE +from ..stoken_handler import filter_by_stoken_and_limit +from ..db_hack import django_db_cleanup_decorator + +from .collection import get_collection, verify_collection_admin + +User = get_typed_user_model() +member_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) +MemberQuerySet = QuerySet[models.CollectionMember] +default_queryset: MemberQuerySet = models.CollectionMember.objects.all() + + +@django_db_cleanup_decorator +def get_queryset(collection: models.Collection = Depends(get_collection)) -> MemberQuerySet: + return default_queryset.filter(collection=collection) + + +@django_db_cleanup_decorator +def get_member(username: str, queryset: MemberQuerySet = Depends(get_queryset)) -> models.CollectionMember: + return get_object_or_404(queryset, user__username__iexact=username) + + +class CollectionMemberModifyAccessLevelIn(BaseModel): + accessLevel: models.AccessLevels + + +class CollectionMemberOut(BaseModel): + username: str + accessLevel: models.AccessLevels + + class Config: + orm_mode = True + + @classmethod + def from_orm(cls: t.Type["CollectionMemberOut"], obj: models.CollectionMember) -> "CollectionMemberOut": + return cls(username=obj.user.username, accessLevel=obj.accessLevel) + + +class MemberListResponse(BaseModel): + data: t.List[CollectionMemberOut] + iterator: t.Optional[str] + done: bool + + +@member_router.get( + "/member/", response_model=MemberListResponse, dependencies=[Depends(verify_collection_admin), *PERMISSIONS_READ] +) +def member_list( + iterator: t.Optional[str] = None, + limit: int = 50, + queryset: MemberQuerySet = Depends(get_queryset), +): + queryset = queryset.order_by("id") + result, new_stoken_obj, done = filter_by_stoken_and_limit( + iterator, limit, queryset, models.CollectionMember.stoken_annotation + ) + new_stoken = new_stoken_obj and new_stoken_obj.uid + + return MemberListResponse( + data=[CollectionMemberOut.from_orm(item) for item in result], + iterator=new_stoken, + done=done, + ) + + +@member_router.delete( + "/member/{username}/", + status_code=status.HTTP_204_NO_CONTENT, + dependencies=[Depends(verify_collection_admin), *PERMISSIONS_READWRITE], +) +def member_delete( + obj: models.CollectionMember = Depends(get_member), +): + obj.revoke() + + +@member_router.patch( + "/member/{username}/", + status_code=status.HTTP_204_NO_CONTENT, + dependencies=[Depends(verify_collection_admin), *PERMISSIONS_READWRITE], +) +def member_patch( + data: CollectionMemberModifyAccessLevelIn, + instance: models.CollectionMember = Depends(get_member), +): + with transaction.atomic(): + # We only allow updating accessLevel + if instance.accessLevel != data.accessLevel: + instance.stoken = models.Stoken.objects.create() + instance.accessLevel = data.accessLevel + instance.save() + + +@member_router.post("/member/leave/", status_code=status.HTTP_204_NO_CONTENT, dependencies=PERMISSIONS_READ) +def member_leave( + user: UserType = Depends(get_authenticated_user), collection: models.Collection = Depends(get_collection) +): + obj = get_object_or_404(collection.members, user=user) + obj.revoke() diff --git a/etebase_fastapi/routers/test_reset_view.py b/etebase_fastapi/routers/test_reset_view.py new file mode 100644 index 0000000..09638e4 --- /dev/null +++ b/etebase_fastapi/routers/test_reset_view.py @@ -0,0 +1,38 @@ +from django.conf import settings +from django.db import transaction +from django.shortcuts import get_object_or_404 +from fastapi import APIRouter, Request, status + +from django_etebase.utils import get_user_queryset, CallbackContext +from .authentication import SignupIn, signup_save +from ..msgpack import MsgpackRoute +from ..exceptions import HttpError +from myauth.models import get_typed_user_model + +test_reset_view_router = APIRouter(route_class=MsgpackRoute, tags=["test helpers"]) +User = get_typed_user_model() + + +@test_reset_view_router.post("/reset/", status_code=status.HTTP_204_NO_CONTENT) +def reset(data: SignupIn, request: Request): + # Only run when in DEBUG mode! It's only used for tests + if not settings.DEBUG: + raise HttpError(code="generic", detail="Only allowed in debug mode.") + + with transaction.atomic(): + user_queryset = get_user_queryset(User.objects.all(), CallbackContext(request.path_params)) + user = get_object_or_404(user_queryset, username=data.user.username) + # Only allow test users for extra safety + if not getattr(user, User.USERNAME_FIELD).startswith("test_user"): + raise HttpError(code="generic", detail="Endpoint not allowed for user.") + + if hasattr(user, "userinfo"): + user.userinfo.delete() + + signup_save(data, request) + # Delete all of the journal data for this user for a clear test env + user.collection_set.all().delete() + user.collectionmember_set.all().delete() + user.incoming_invitations.all().delete() + + # FIXME: also delete chunk files!!! diff --git a/etebase_fastapi/routers/websocket.py b/etebase_fastapi/routers/websocket.py new file mode 100644 index 0000000..3fc535f --- /dev/null +++ b/etebase_fastapi/routers/websocket.py @@ -0,0 +1,145 @@ +import asyncio +import typing as t + +import aioredis +from asgiref.sync import sync_to_async +from django.db.models import QuerySet +from fastapi import APIRouter, Depends, WebSocket, WebSocketDisconnect, status +import nacl.encoding +import nacl.utils + +from django_etebase import models +from django_etebase.utils import CallbackContext, get_user_queryset +from myauth.models import UserType, get_typed_user_model + +from ..dependencies import get_collection_queryset, get_item_queryset +from ..exceptions import NotSupported +from ..msgpack import MsgpackRoute, msgpack_decode, msgpack_encode +from ..redis import redisw +from ..utils import BaseModel, permission_responses + + +User = get_typed_user_model() +websocket_router = APIRouter(route_class=MsgpackRoute, responses=permission_responses) +CollectionQuerySet = QuerySet[models.Collection] + + +TICKET_VALIDITY_SECONDS = 10 + + +class TicketRequest(BaseModel): + collection: str + + +class TicketOut(BaseModel): + ticket: str + + +class TicketInner(BaseModel): + user: int + req: TicketRequest + + +async def get_ticket( + ticket_request: TicketRequest, + user: UserType, +): + """Get an authentication ticket that can be used with the websocket endpoint for authentication""" + if not redisw.is_active: + raise NotSupported(detail="This end-point requires Redis to be configured") + + uid = nacl.encoding.URLSafeBase64Encoder.encode(nacl.utils.random(32)) + ticket_model = TicketInner(user=user.id, req=ticket_request) + ticket_raw = msgpack_encode(ticket_model.dict()) + await redisw.redis.set(uid, ticket_raw, expire=TICKET_VALIDITY_SECONDS * 1000) + return TicketOut(ticket=uid) + + +async def load_websocket_ticket(websocket: WebSocket, ticket: str) -> t.Optional[TicketInner]: + content = await redisw.redis.get(ticket) + if content is None: + await websocket.close(code=status.WS_1008_POLICY_VIOLATION) + return None + await redisw.redis.delete(ticket) + return TicketInner(**msgpack_decode(content)) + + +def get_websocket_user(websocket: WebSocket, ticket_model: t.Optional[TicketInner] = Depends(load_websocket_ticket)): + if ticket_model is None: + return None + user_queryset = get_user_queryset(User.objects.all(), CallbackContext(websocket.path_params)) + return user_queryset.get(id=ticket_model.user) + + +@websocket_router.websocket("/{ticket}/") +async def websocket_endpoint( + websocket: WebSocket, + stoken: t.Optional[str] = None, + user: t.Optional[UserType] = Depends(get_websocket_user), + ticket_model: TicketInner = Depends(load_websocket_ticket), +): + if user is None: + return + await websocket.accept() + await redis_connector(websocket, ticket_model, user, stoken) + + +async def send_item_updates( + websocket: WebSocket, + collection: models.Collection, + user: UserType, + stoken: t.Optional[str], +): + from .collection import item_list_common + + done = False + while not done: + queryset = await sync_to_async(get_item_queryset)(collection) + response = await sync_to_async(item_list_common)(queryset, user, stoken, limit=50, prefetch="auto") + done = response.done + if len(response.data) > 0: + await websocket.send_bytes(msgpack_encode(response.dict())) + + +async def redis_connector(websocket: WebSocket, ticket_model: TicketInner, user: UserType, stoken: t.Optional[str]): + async def producer_handler(r: aioredis.Redis, ws: WebSocket): + channel_name = f"col.{ticket_model.req.collection}" + (channel,) = await r.psubscribe(channel_name) + assert isinstance(channel, aioredis.Channel) + + # Send missing items if we are not up to date + queryset: QuerySet[models.Collection] = get_collection_queryset(user) + collection: t.Optional[models.Collection] = await sync_to_async( + queryset.filter(uid=ticket_model.req.collection).first + )() + if collection is None: + await websocket.close(code=status.WS_1008_POLICY_VIOLATION) + return + await send_item_updates(websocket, collection, user, stoken) + + try: + while True: + # We wait on the websocket so we fail if web sockets fail or get data + receive = asyncio.create_task(websocket.receive()) + done, pending = await asyncio.wait( + {receive, channel.wait_message()}, return_when=asyncio.FIRST_COMPLETED + ) + for task in pending: + task.cancel() + if receive in done: + # Web socket should never receieve any data + await websocket.close(code=status.WS_1008_POLICY_VIOLATION) + return + + message_raw = t.cast(t.Optional[t.Tuple[str, bytes]], await channel.get()) + if message_raw: + _, message = message_raw + await ws.send_bytes(message) + + except aioredis.errors.ConnectionClosedError: + await websocket.close(code=status.WS_1012_SERVICE_RESTART) + except WebSocketDisconnect: + pass + + redis = redisw.redis + await producer_handler(redis, websocket) diff --git a/etebase_fastapi/sendfile/LICENSE b/etebase_fastapi/sendfile/LICENSE new file mode 100644 index 0000000..4b733c8 --- /dev/null +++ b/etebase_fastapi/sendfile/LICENSE @@ -0,0 +1,28 @@ +Copyright (c) 2011, Sensible Development. +Copyright (c) 2019, Matt Molyneaux +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + 1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + + 3. Neither the name of Django Send File nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/etebase_fastapi/sendfile/README.md b/etebase_fastapi/sendfile/README.md new file mode 100644 index 0000000..aab5091 --- /dev/null +++ b/etebase_fastapi/sendfile/README.md @@ -0,0 +1,3 @@ +Heavily inspired + code borrowed from: https://github.com/moggers87/django-sendfile2/ + +We just simplified and inlined it because we don't want another external dependency for distribution packagers to package, as well as need a much simpler version. diff --git a/etebase_fastapi/sendfile/__init__.py b/etebase_fastapi/sendfile/__init__.py new file mode 100644 index 0000000..4949aa5 --- /dev/null +++ b/etebase_fastapi/sendfile/__init__.py @@ -0,0 +1 @@ +from .utils import sendfile # noqa diff --git a/django_etebase/token_auth/admin.py b/etebase_fastapi/sendfile/backends/__init__.py similarity index 100% rename from django_etebase/token_auth/admin.py rename to etebase_fastapi/sendfile/backends/__init__.py diff --git a/etebase_fastapi/sendfile/backends/mod_wsgi.py b/etebase_fastapi/sendfile/backends/mod_wsgi.py new file mode 100644 index 0000000..b8fc6c0 --- /dev/null +++ b/etebase_fastapi/sendfile/backends/mod_wsgi.py @@ -0,0 +1,9 @@ +from __future__ import absolute_import + +from fastapi import Response + +from ..utils import _convert_file_to_url + + +def sendfile(filename, **kwargs): + return Response(headers={"Location": _convert_file_to_url(filename)}) diff --git a/etebase_fastapi/sendfile/backends/nginx.py b/etebase_fastapi/sendfile/backends/nginx.py new file mode 100644 index 0000000..b22e0d0 --- /dev/null +++ b/etebase_fastapi/sendfile/backends/nginx.py @@ -0,0 +1,9 @@ +from __future__ import absolute_import + +from fastapi import Response + +from ..utils import _convert_file_to_url + + +def sendfile(filename, **kwargs): + return Response(headers={"X-Accel-Redirect": _convert_file_to_url(filename)}) diff --git a/etebase_fastapi/sendfile/backends/simple.py b/etebase_fastapi/sendfile/backends/simple.py new file mode 100644 index 0000000..f3a3548 --- /dev/null +++ b/etebase_fastapi/sendfile/backends/simple.py @@ -0,0 +1,12 @@ +from fastapi.responses import FileResponse + + +def sendfile(filename, mimetype, **kwargs): + """Use the SENDFILE_ROOT value composed with the path arrived as argument + to build an absolute path with which resolve and return the file contents. + + If the path points to a file out of the root directory (should cover both + situations with '..' and symlinks) then a 404 is raised. + """ + + return FileResponse(filename, media_type=mimetype) diff --git a/etebase_fastapi/sendfile/backends/xsendfile.py b/etebase_fastapi/sendfile/backends/xsendfile.py new file mode 100644 index 0000000..530f6a1 --- /dev/null +++ b/etebase_fastapi/sendfile/backends/xsendfile.py @@ -0,0 +1,6 @@ +from fastapi import Response + + +def sendfile(filename, **kwargs): + filename = str(filename) + return Response(headers={"X-Sendfile": filename}) diff --git a/etebase_fastapi/sendfile/utils.py b/etebase_fastapi/sendfile/utils.py new file mode 100644 index 0000000..c35d6df --- /dev/null +++ b/etebase_fastapi/sendfile/utils.py @@ -0,0 +1,88 @@ +from functools import lru_cache +from importlib import import_module +from pathlib import Path, PurePath +from urllib.parse import quote +import logging + +from fastapi import status +from ..exceptions import HttpError + +from django.conf import settings +from django.core.exceptions import ImproperlyConfigured + +logger = logging.getLogger(__name__) + + +@lru_cache(maxsize=None) +def _get_sendfile(): + backend = getattr(settings, "SENDFILE_BACKEND", None) + if not backend: + raise ImproperlyConfigured("You must specify a value for SENDFILE_BACKEND") + module = import_module(backend) + return module.sendfile + + +def _convert_file_to_url(path): + try: + url_root = PurePath(getattr(settings, "SENDFILE_URL", None)) + except TypeError: + return path + + path_root = PurePath(settings.SENDFILE_ROOT) + path_obj = PurePath(path) + + relpath = path_obj.relative_to(path_root) + # Python 3.5: Path.resolve() has no `strict` kwarg, so use pathmod from an + # already instantiated Path object + url = relpath._flavour.pathmod.normpath(str(url_root / relpath)) + + return quote(str(url)) + + +def _sanitize_path(filepath): + try: + path_root = Path(getattr(settings, "SENDFILE_ROOT", None)) + except TypeError: + raise ImproperlyConfigured("You must specify a value for SENDFILE_ROOT") + + filepath_obj = Path(filepath) + + # get absolute path + # Python 3.5: Path.resolve() has no `strict` kwarg, so use pathmod from an + # already instantiated Path object + filepath_abs = Path(filepath_obj._flavour.pathmod.normpath(str(path_root / filepath_obj))) + + # if filepath_abs is not relative to path_root, relative_to throws an error + try: + filepath_abs.relative_to(path_root) + except ValueError: + raise HttpError( + "generic", "{} wrt {} is impossible".format(filepath_abs, path_root), status_code=status.HTTP_404_NOT_FOUND + ) + + return filepath_abs + + +def sendfile(filename, mimetype="application/octet-stream", encoding=None): + """ + Create a response to send file using backend configured in ``SENDFILE_BACKEND`` + + ``filename`` is the absolute path to the file to send. + """ + filepath_obj = _sanitize_path(filename) + logger.debug( + "filename '%s' requested \"\ + \"-> filepath '%s' obtained", + filename, + filepath_obj, + ) + _sendfile = _get_sendfile() + + if not filepath_obj.exists(): + raise HttpError("does_not_exist", '"%s" does not exist' % filepath_obj, status_code=status.HTTP_404_NOT_FOUND) + + response = _sendfile(filepath_obj, mimetype=mimetype) + + response.headers["Content-Type"] = mimetype + + return response diff --git a/etebase_fastapi/stoken_handler.py b/etebase_fastapi/stoken_handler.py new file mode 100644 index 0000000..b6f2999 --- /dev/null +++ b/etebase_fastapi/stoken_handler.py @@ -0,0 +1,62 @@ +import typing as t + +from django.db.models import QuerySet +from fastapi import status + +from django_etebase.models import Stoken + +from .exceptions import HttpError + +# TODO missing stoken_annotation type +StokenAnnotation = t.Any + + +def get_stoken_obj(stoken: t.Optional[str]) -> t.Optional[Stoken]: + if stoken: + try: + return Stoken.objects.get(uid=stoken) + except Stoken.DoesNotExist: + raise HttpError("bad_stoken", "Invalid stoken.", status_code=status.HTTP_400_BAD_REQUEST) + + return None + + +def filter_by_stoken( + stoken: t.Optional[str], queryset: QuerySet, stoken_annotation: StokenAnnotation +) -> t.Tuple[QuerySet, t.Optional[Stoken]]: + stoken_rev = get_stoken_obj(stoken) + + queryset = queryset.annotate(max_stoken=stoken_annotation).order_by("max_stoken") + + if stoken_rev is not None: + queryset = queryset.filter(max_stoken__gt=stoken_rev.id) + + return queryset, stoken_rev + + +def get_queryset_stoken(queryset: t.Iterable[t.Any]) -> t.Optional[Stoken]: + maxid = -1 + for row in queryset: + rowmaxid = getattr(row, "max_stoken") or -1 + maxid = max(maxid, rowmaxid) + new_stoken = Stoken.objects.get(id=maxid) if (maxid >= 0) else None + + return new_stoken or None + + +def filter_by_stoken_and_limit( + stoken: t.Optional[str], limit: int, queryset: QuerySet, stoken_annotation: StokenAnnotation +) -> t.Tuple[list, t.Optional[Stoken], bool]: + + queryset, stoken_rev = filter_by_stoken(stoken=stoken, queryset=queryset, stoken_annotation=stoken_annotation) + + result = list(queryset[: limit + 1]) + if len(result) < limit + 1: + done = True + else: + done = False + result = result[:-1] + + new_stoken_obj = get_queryset_stoken(result) or stoken_rev + + return result, new_stoken_obj, done diff --git a/etebase_fastapi/utils.py b/etebase_fastapi/utils.py new file mode 100644 index 0000000..09c223e --- /dev/null +++ b/etebase_fastapi/utils.py @@ -0,0 +1,85 @@ +import dataclasses +import typing as t +from typing_extensions import Literal +import msgpack +import base64 + +from fastapi import status, Query, Depends +from pydantic import BaseModel as PyBaseModel + +from django.db.models import Model, QuerySet +from django.core.exceptions import ObjectDoesNotExist + +from django_etebase import app_settings +from django_etebase.models import AccessLevels +from myauth.models import UserType, get_typed_user_model + +from .exceptions import HttpError, HttpErrorOut + +User = get_typed_user_model() + +Prefetch = Literal["auto", "medium"] +PrefetchQuery = Query(default="auto") + + +T = t.TypeVar("T", bound=Model, covariant=True) + + +class BaseModel(PyBaseModel): + class Config: + json_encoders = { + bytes: lambda x: x, + } + + +@dataclasses.dataclass +class Context: + user: t.Optional[UserType] + prefetch: t.Optional[Prefetch] + + +def get_object_or_404(queryset: QuerySet[T], **kwargs) -> T: + try: + return queryset.get(**kwargs) + except ObjectDoesNotExist as e: + raise HttpError("does_not_exist", str(e), status_code=status.HTTP_404_NOT_FOUND) + + +def is_collection_admin(collection, user): + member = collection.members.filter(user=user).first() + return (member is not None) and (member.accessLevel == AccessLevels.ADMIN) + + +def msgpack_encode(content) -> bytes: + ret = msgpack.packb(content, use_bin_type=True) + assert ret is not None + return ret + + +def msgpack_decode(content: bytes): + return msgpack.unpackb(content, raw=False) + + +def b64encode(value: bytes): + return base64.urlsafe_b64encode(value).decode("ascii").strip("=") + + +def b64decode(data: str): + data += "=" * ((4 - len(data) % 4) % 4) + return base64.urlsafe_b64decode(data) + + +def get_user_username_email_kwargs(username: str): + field_name = User.EMAIL_FIELD if "@" in username else User.USERNAME_FIELD + return {field_name + "__iexact": username.lower()} + + +PERMISSIONS_READ = [Depends(x) for x in app_settings.API_PERMISSIONS_READ] +PERMISSIONS_READWRITE = PERMISSIONS_READ + [Depends(x) for x in app_settings.API_PERMISSIONS_WRITE] + + +response_model_dict = {"model": HttpErrorOut} +permission_responses: t.Dict[t.Union[int, str], t.Dict[str, t.Any]] = { + 401: response_model_dict, + 403: response_model_dict, +} diff --git a/etebase_server/asgi.py b/etebase_server/asgi.py index 0bf63ec..25dbf77 100644 --- a/etebase_server/asgi.py +++ b/etebase_server/asgi.py @@ -1,16 +1,19 @@ -""" -ASGI config for etebase_server project. - -It exposes the ASGI callable as a module-level variable named ``application``. - -For more information on this file, see -https://docs.djangoproject.com/en/3.0/howto/deployment/asgi/ -""" - import os from django.core.asgi import get_asgi_application os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") +django_application = get_asgi_application() -application = get_asgi_application() + +def create_application(): + from etebase_fastapi.main import create_application + + app = create_application() + + app.mount("/", django_application) + + return app + + +application = create_application() diff --git a/etebase_server/settings.py b/etebase_server/settings.py index e9a4903..8ace7bb 100644 --- a/etebase_server/settings.py +++ b/etebase_server/settings.py @@ -53,8 +53,6 @@ INSTALLED_APPS = [ "django.contrib.sessions", "django.contrib.messages", "django.contrib.staticfiles", - "corsheaders", - "rest_framework", "myauth.apps.MyauthConfig", "django_etebase.apps.DjangoEtebaseConfig", "django_etebase.token_auth.apps.TokenAuthConfig", @@ -63,7 +61,6 @@ INSTALLED_APPS = [ MIDDLEWARE = [ "django.middleware.security.SecurityMiddleware", "django.contrib.sessions.middleware.SessionMiddleware", - "corsheaders.middleware.CorsMiddleware", "django.middleware.common.CommonMiddleware", "django.middleware.csrf.CsrfViewMiddleware", "django.contrib.auth.middleware.AuthenticationMiddleware", @@ -124,9 +121,6 @@ USE_L10N = True USE_TZ = True -# Cors -CORS_ORIGIN_ALLOW_ALL = True - # Static files (CSS, JavaScript, Images) # https://docs.djangoproject.com/en/3.0/howto/static-files/ @@ -141,7 +135,6 @@ ETEBASE_API_AUTHENTICATORS = ( "django_etebase.token_auth.authentication.TokenAuthentication", "rest_framework.authentication.SessionAuthentication", ) -ETEBASE_CREATE_USER_FUNC = "django_etebase.utils.create_user_blocked" # Define where to find configuration files config_locations = [ @@ -166,6 +159,9 @@ if any(os.path.isfile(x) for x in config_locations): TIME_ZONE = section.get("time_zone", TIME_ZONE) DEBUG = section.getboolean("debug", DEBUG) + if "redis_uri" in section: + ETEBASE_REDIS_URI = section.get("redis_uri") + if "allowed_hosts" in config: ALLOWED_HOSTS = [y for x, y in config.items("allowed_hosts")] @@ -184,6 +180,12 @@ if any(os.path.isfile(x) for x in config_locations): ETEBASE_CREATE_USER_FUNC = "myauth.ldap.create_user" ETEBASE_API_PERMISSIONS.append("myauth.ldap.LDAPUserExists") +ETEBASE_CREATE_USER_FUNC = "django_etebase.utils.create_user_blocked" + +# Efficient file streaming (for large files) +SENDFILE_BACKEND = "django_etebase.sendfile.backends.simple" +SENDFILE_ROOT = MEDIA_URL + # Make an `etebase_server_settings` module available to override settings. try: from etebase_server_settings import * diff --git a/etebase_server/urls.py b/etebase_server/urls.py index f285977..7cf5a60 100644 --- a/etebase_server/urls.py +++ b/etebase_server/urls.py @@ -1,16 +1,25 @@ +import os + from django.conf import settings -from django.conf.urls import include, url +from django.conf.urls import url from django.contrib import admin -from django.urls import path +from django.urls import path, re_path from django.views.generic import TemplateView +from django.views.static import serve +from django.contrib.staticfiles import finders urlpatterns = [ - url(r"^api/", include("django_etebase.urls")), url(r"^admin/", admin.site.urls), path("", TemplateView.as_view(template_name="success.html")), ] if settings.DEBUG: - urlpatterns += [ - url(r"^api-auth/", include("rest_framework.urls", namespace="rest_framework")), - ] + + def serve_static(request, path): + filename = finders.find(path) + dirname = os.path.dirname(filename) + basename = os.path.basename(filename) + + return serve(request, basename, dirname) + + urlpatterns += [re_path(r"^static/(?P.*)$", serve_static)] diff --git a/etebase_server/wsgi.py b/etebase_server/wsgi.py deleted file mode 100644 index 908f88c..0000000 --- a/etebase_server/wsgi.py +++ /dev/null @@ -1,16 +0,0 @@ -""" -WSGI config for etebase_server project. - -It exposes the WSGI callable as a module-level variable named ``application``. - -For more information on this file, see -https://docs.djangoproject.com/en/3.0/howto/deployment/wsgi/ -""" - -import os - -from django.core.wsgi import get_wsgi_application - -os.environ.setdefault("DJANGO_SETTINGS_MODULE", "etebase_server.settings") - -application = get_wsgi_application() diff --git a/myauth/forms.py b/myauth/forms.py index 7aacb9b..fc2be74 100644 --- a/myauth/forms.py +++ b/myauth/forms.py @@ -1,8 +1,8 @@ from django import forms -from django.contrib.auth import get_user_model from django.contrib.auth.forms import UsernameField +from myauth.models import get_typed_user_model -User = get_user_model() +User = get_typed_user_model() class AdminUserCreationForm(forms.ModelForm): diff --git a/myauth/migrations/0003_auto_20201119_0810.py b/myauth/migrations/0003_auto_20201119_0810.py new file mode 100644 index 0000000..cfd2ec3 --- /dev/null +++ b/myauth/migrations/0003_auto_20201119_0810.py @@ -0,0 +1,37 @@ +# Generated by Django 3.1.1 on 2020-11-19 08:10 + +from django.db import migrations, models +import myauth.models + + +class Migration(migrations.Migration): + + dependencies = [ + ("myauth", "0002_auto_20200515_0801"), + ] + + operations = [ + migrations.AlterModelManagers( + name="user", + managers=[ + ("objects", myauth.models.UserManager()), + ], + ), + migrations.AlterField( + model_name="user", + name="first_name", + field=models.CharField(blank=True, max_length=150, verbose_name="first name"), + ), + migrations.AlterField( + model_name="user", + name="username", + field=models.CharField( + error_messages={"unique": "A user with that username already exists."}, + help_text="Required. 150 characters or fewer. Letters, digits and ./-/_ only.", + max_length=150, + unique=True, + validators=[myauth.models.UnicodeUsernameValidator()], + verbose_name="username", + ), + ), + ] diff --git a/myauth/models.py b/myauth/models.py index d6585a8..89b94b4 100644 --- a/myauth/models.py +++ b/myauth/models.py @@ -1,3 +1,5 @@ +import typing as t + from django.contrib.auth.models import AbstractUser, UserManager as DjangoUserManager from django.core import validators from django.db import models @@ -13,14 +15,15 @@ class UnicodeUsernameValidator(validators.RegexValidator): class UserManager(DjangoUserManager): - def get_by_natural_key(self, username): + def get_by_natural_key(self, username: str): return self.get(**{self.model.USERNAME_FIELD + "__iexact": username}) class User(AbstractUser): + id: int username_validator = UnicodeUsernameValidator() - objects = UserManager() + objects: UserManager = UserManager() username = models.CharField( _("username"), @@ -28,9 +31,21 @@ class User(AbstractUser): unique=True, help_text=_("Required. 150 characters or fewer. Letters, digits and ./-/_ only."), validators=[username_validator], - error_messages={"unique": _("A user with that username already exists."),}, + error_messages={ + "unique": _("A user with that username already exists."), + }, ) @classmethod - def normalize_username(cls, username): + def normalize_username(cls, username: str): return super().normalize_username(username).lower() + + +UserType = User + + +def get_typed_user_model() -> UserType: + from django.contrib.auth import get_user_model + + ret: t.Any = get_user_model() + return ret diff --git a/mypy.ini b/mypy.ini new file mode 100644 index 0000000..0a96c4c --- /dev/null +++ b/mypy.ini @@ -0,0 +1,5 @@ +[mypy] +plugins = mypy_django_plugin.main + +[mypy.plugins.django-stubs] +django_settings_module = "etebase_server.settings" diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000..15a8b60 --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,28 @@ +# +# This file is autogenerated by pip-compile +# To update, run: +# +# pip-compile --output-file=requirements-dev.txt requirements.in/development.txt +# +appdirs==1.4.4 # via black +asgiref==3.3.1 # via django +black==20.8b1 # via -r requirements.in/development.txt +click==7.1.2 # via black, pip-tools +coverage==5.3.1 # via -r requirements.in/development.txt +django-stubs==1.7.0 # via -r requirements.in/development.txt +django==3.1.4 # via django-stubs +mypy-extensions==0.4.3 # via black, mypy +mypy==0.790 # via django-stubs +pathspec==0.8.1 # via black +pip-tools==5.4.0 # via -r requirements.in/development.txt +pytz==2020.5 # via django +pywatchman==1.4.1 # via -r requirements.in/development.txt +regex==2020.11.13 # via black +six==1.15.0 # via pip-tools +sqlparse==0.4.1 # via django +toml==0.10.2 # via black +typed-ast==1.4.1 # via black, mypy +typing-extensions==3.7.4.3 # via black, django-stubs, mypy + +# The following packages are considered to be unsafe in a requirements file: +# pip diff --git a/requirements.in/base.txt b/requirements.in/base.txt index 7d5bf7e..4b9b85b 100644 --- a/requirements.in/base.txt +++ b/requirements.in/base.txt @@ -1,7 +1,8 @@ django -django-cors-headers -djangorestframework -drf-nested-routers msgpack -psycopg2-binary pynacl +fastapi +typing_extensions +uvicorn[standard] +aiofiles +aioredis diff --git a/requirements.in/development.txt b/requirements.in/development.txt index a956471..fb281d3 100644 --- a/requirements.in/development.txt +++ b/requirements.in/development.txt @@ -1,4 +1,5 @@ coverage pip-tools pywatchman -black \ No newline at end of file +black +django-stubs diff --git a/requirements.txt b/requirements.txt index f6c8ed4..213462f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,16 +4,29 @@ # # pip-compile --output-file=requirements.txt requirements.in/base.txt # -asgiref==3.2.10 # via django -cffi==1.14.0 # via pynacl -django-cors-headers==3.2.1 # via -r requirements.in/base.txt -django==3.1.1 # via -r requirements.in/base.txt, django-cors-headers, djangorestframework, drf-nested-routers -djangorestframework==3.11.0 # via -r requirements.in/base.txt, drf-nested-routers -drf-nested-routers==0.91 # via -r requirements.in/base.txt -msgpack==1.0.0 # via -r requirements.in/base.txt -psycopg2-binary==2.8.4 # via -r requirements.in/base.txt +aiofiles==0.6.0 # via -r requirements.in/base.txt +aioredis==1.3.1 # via -r requirements.in/base.txt +asgiref==3.3.1 # via django +async-timeout==3.0.1 # via aioredis +cffi==1.14.4 # via pynacl +click==7.1.2 # via uvicorn +django==3.1.4 # via -r requirements.in/base.txt +fastapi==0.63.0 # via -r requirements.in/base.txt +h11==0.11.0 # via uvicorn +hiredis==1.1.0 # via aioredis +httptools==0.1.1 # via uvicorn +msgpack==1.0.2 # via -r requirements.in/base.txt pycparser==2.20 # via cffi -pynacl==1.3.0 # via -r requirements.in/base.txt -pytz==2019.3 # via django -six==1.14.0 # via pynacl -sqlparse==0.3.0 # via django +pydantic==1.7.3 # via fastapi +pynacl==1.4.0 # via -r requirements.in/base.txt +python-dotenv==0.15.0 # via uvicorn +pytz==2020.4 # via django +pyyaml==5.3.1 # via uvicorn +six==1.15.0 # via pynacl +sqlparse==0.4.1 # via django +starlette==0.13.6 # via fastapi +typing-extensions==3.7.4.3 # via -r requirements.in/base.txt +uvicorn[standard]==0.13.2 # via -r requirements.in/base.txt +uvloop==0.14.0 # via uvicorn +watchgod==0.6 # via uvicorn +websockets==8.1 # via uvicorn