ITW: Overly secure
serverside storage
I need a file server.
One that I can access from any browser with just a username / password.


There are services that offer this already, but they all have some problems:
 *
They cost money.
 *
Have a storage limit.
 *
Rate limiting.
 *
Their apis (if they have any) are slow.
 *
Don't work offline.
 *
Aren't open source.
 *
Have sketchy security.

So I'll just have to make my own (yay), so what features do I want?
 *
Open source (obviously).
 *
Runs locally (which also means it can work offline).
 *
Fast apis.
 *
Trustless security.

What do I mean by trustless security?
I mean a system where the server can be a bad actor and not get your data, where man in the middle attacks do nothing, where the files never leave the client unencrypted.

Part 1:
Serving the server
|[click to close]

To create the server I used nodejs (or bun to be specific), bun has a really nice easy to use server api, so it'll handle all the server stuff easily.

Part 2:
Messaging the server
|[click to close]

Now this could be easy, using https I would know that my messages aren't being messed with, but if I want to host this server myself I'd rather not have to set up the needed certificates.
I've actually already done this before with some really cool math magic that is the Diffie-Hellman key exchange, it has to peers exchange public keys and use them to generate a shared key used for symmetric message encryption (illustration below).


Part 3:
Proving identity to the server
|[click to close]

I can hardly just have any client able to claim to be any user, that would allow anyone full access to your files (which would do them no good, since they're still encrypted remember?), but more importantly it would allow them to delete your files / fill up your storage with junk files, so the server needs some way to know the client is who they say they are.
One common way is to send the server your login info, but that isn't an option when you don't trust the server, the secret you send it has to only matter to it and be unusable to everyone else, one idea would be for the server to send a one-time code to new users that they remember and use as part of their login, but that would require them to write it down somewhere for each server they use.
But with a little modifications that actually does work well, the server sends a "seed" to the client, who then hashes it + their login info to create a token that they send back to the server, if this is the first time the server has connected to this user it stores the token as their auth token, and if it's not the first time it compares the sent token to the stored token (illustrations below).


Part 4:
Defining the api
|[click to close]

Now that I have a server, and can safely message it without listening ears, and it knows who I am and won't let random people mess with my stuff, what do I need it to do? Since I don't want the server to know what my files are, and as little information about them as possible, I want a bare bones api that can only do low level stuff, reading, writing, and deleting files in a flat folder for the user.

Part 5:
Using the api
|[click to close]

I can now use the server to securely save files, but I don't want the server to be able to read them, so I need to encrypt them, the problem is a user's username + password + seed alone isn't enough data to create a secure key.
My first thought was to somehow have the server store the key in a way that it can't use it, but I wasn't able to find a way that that could work.
Unless... turns out I can derive an AES key from a small string, and then use that base key + some extra random data per use to create a proper key, each file will have the used "salt" and "iv" stored with it in plaintext, but for once that's actually fine, and then they can be used to recreate the same key later to decrypt the file (illustrations below).


Part 6:
[BONUS] Adding a virtual filesystem
|[click to close]

Now that everything's done it's time to use it, how about a basic file storage system, ok the user uploaded some images nice, and some text files...
oh now they want all of their images back, but I never recorded what was what, I don't even have a list of the files they have saved...
A slightly more extensive filesystem would do wonders, but making api calls to the server for every filesystem command is way too inefficient, so it needs to be a virtual filesystem.
I can do this by having a main "index" file that stores the state of the filesystem, I just have to keep it updated, but doing this allows for a lot more features:
 *
Folders.
 *
Reading a folder's contents.
 *
Automatic deduplication of files.
 * *
It stores the hash of each saved file in the index.
 * *
When I delete a file it only deletes the real file if no other file in the index has the same hash.
 * *
Whenever I save a file it first sees if it already have an identical file saved, if so it can skip saving the actual file.
 *
Auto creating folders as needed.
 *
Auto deleting empty folders.
 *
I can see errors / warning for overwriting a file, trying to read a file that doesn't exist etc before actually messaging the server.
 *
Faster queries (since it doesn't have to query the server for each operation).
That's all for this project for now, if you want to see any of the code you can see it on my
github.
Return to top