mirror of https://github.com/usememos/memos.git
310 lines
8.7 KiB
Markdown
310 lines
8.7 KiB
Markdown
# Fileserver Package
|
|
|
|
## Overview
|
|
|
|
The `fileserver` package handles all binary file serving for Memos using native HTTP handlers. It was created to replace gRPC-based binary serving, which had limitations with HTTP range requests (required for Safari video/audio playback).
|
|
|
|
## Responsibilities
|
|
|
|
- Serve attachment binary files (images, videos, audio, documents)
|
|
- Serve user avatar images
|
|
- Handle HTTP range requests for video/audio streaming
|
|
- Authenticate requests using session cookies or JWT tokens
|
|
- Check permissions for private content
|
|
- Generate and serve image thumbnails
|
|
- Prevent XSS attacks on uploaded content
|
|
- Support S3 external storage
|
|
|
|
## Architecture
|
|
|
|
### Design Principles
|
|
|
|
1. **Separation of Concerns**: Binary files via HTTP, metadata via gRPC
|
|
2. **DRY**: Imports auth constants from `api/v1` package (single source of truth)
|
|
3. **Security First**: Authentication, authorization, and XSS prevention
|
|
4. **Performance**: Native HTTP streaming with proper caching headers
|
|
|
|
### Package Structure
|
|
|
|
```
|
|
fileserver/
|
|
├── fileserver.go # Main service and HTTP handlers
|
|
├── README.md # This file
|
|
└── fileserver_test.go # Tests (to be added)
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
### 1. Attachment Binary
|
|
```
|
|
GET /file/attachments/:uid/:filename[?thumbnail=true]
|
|
```
|
|
|
|
**Parameters:**
|
|
- `uid` - Attachment unique identifier
|
|
- `filename` - Original filename
|
|
- `thumbnail` (optional) - Return thumbnail for images
|
|
|
|
**Authentication:** Required for non-public memos
|
|
|
|
**Response:**
|
|
- `200 OK` - File content with proper Content-Type
|
|
- `206 Partial Content` - For range requests (video/audio)
|
|
- `401 Unauthorized` - Authentication required
|
|
- `403 Forbidden` - User not authorized
|
|
- `404 Not Found` - Attachment not found
|
|
|
|
**Headers:**
|
|
- `Content-Type` - MIME type of the file
|
|
- `Cache-Control: public, max-age=3600`
|
|
- `Accept-Ranges: bytes` - For video/audio
|
|
- `Content-Range` - For partial responses (206)
|
|
|
|
### 2. User Avatar
|
|
```
|
|
GET /file/users/:identifier/avatar
|
|
```
|
|
|
|
**Parameters:**
|
|
- `identifier` - User ID (e.g., `1`) or username (e.g., `steven`)
|
|
|
|
**Authentication:** Not required (avatars are public)
|
|
|
|
**Response:**
|
|
- `200 OK` - Avatar image (PNG/JPEG)
|
|
- `404 Not Found` - User not found or no avatar set
|
|
|
|
**Headers:**
|
|
- `Content-Type` - image/png or image/jpeg
|
|
- `Cache-Control: public, max-age=3600`
|
|
|
|
## Authentication
|
|
|
|
### Supported Methods
|
|
|
|
The fileserver supports two authentication methods, checked in order:
|
|
|
|
1. **Session Cookie** (`user_session`)
|
|
- Cookie format: `{userID}-{sessionID}`
|
|
- Validates session exists and hasn't expired (14-day sliding window)
|
|
- Updates last accessed time on success
|
|
|
|
2. **JWT Bearer Token** (`Authorization: Bearer {token}`)
|
|
- Validates JWT signature using server secret
|
|
- Checks token exists in user's access tokens (for revocation)
|
|
- Extracts user ID from token claims
|
|
|
|
### Authentication Flow
|
|
|
|
```
|
|
Request → getCurrentUser()
|
|
├─→ Try Session Cookie
|
|
│ ├─→ Parse cookie value
|
|
│ ├─→ Get user from DB
|
|
│ ├─→ Validate session
|
|
│ └─→ Return user (if valid)
|
|
│
|
|
└─→ Try JWT Token
|
|
├─→ Parse Authorization header
|
|
├─→ Verify JWT signature
|
|
├─→ Get user from DB
|
|
├─→ Validate token in access tokens list
|
|
└─→ Return user (if valid)
|
|
```
|
|
|
|
### Permission Model
|
|
|
|
**Attachments:**
|
|
- Unlinked: Public (no auth required)
|
|
- Public memo: Public (no auth required)
|
|
- Protected memo: Requires authentication
|
|
- Private memo: Creator only
|
|
|
|
**Avatars:**
|
|
- Always public (no auth required)
|
|
|
|
## Key Functions
|
|
|
|
### HTTP Handlers
|
|
|
|
#### `serveAttachmentFile(c echo.Context) error`
|
|
Main handler for attachment binary serving.
|
|
|
|
**Flow:**
|
|
1. Extract UID from URL parameter
|
|
2. Fetch attachment from database
|
|
3. Check permissions (memo visibility)
|
|
4. Get binary blob (local file, S3, or database)
|
|
5. Handle thumbnail request (if applicable)
|
|
6. Set security headers (XSS prevention)
|
|
7. Serve with range request support (video/audio)
|
|
|
|
#### `serveUserAvatar(c echo.Context) error`
|
|
Main handler for user avatar serving.
|
|
|
|
**Flow:**
|
|
1. Extract identifier (ID or username) from URL
|
|
2. Lookup user in database
|
|
3. Check if avatar exists
|
|
4. Decode base64 data URI
|
|
5. Serve with proper content type and caching
|
|
|
|
### Authentication
|
|
|
|
#### `getCurrentUser(ctx, c) (*store.User, error)`
|
|
Authenticates request using session cookie or JWT token.
|
|
|
|
#### `authenticateBySession(ctx, cookie) (*store.User, error)`
|
|
Validates session cookie and returns authenticated user.
|
|
|
|
#### `authenticateByJWT(ctx, token) (*store.User, error)`
|
|
Validates JWT access token and returns authenticated user.
|
|
|
|
### Permission Checks
|
|
|
|
#### `checkAttachmentPermission(ctx, c, attachment) error`
|
|
Validates user has permission to access attachment based on memo visibility.
|
|
|
|
### File Operations
|
|
|
|
#### `getAttachmentBlob(attachment) ([]byte, error)`
|
|
Retrieves binary content from local storage, S3, or database.
|
|
|
|
#### `getOrGenerateThumbnail(ctx, attachment) ([]byte, error)`
|
|
Returns cached thumbnail or generates new one (with semaphore limiting).
|
|
|
|
### Utilities
|
|
|
|
#### `getUserByIdentifier(ctx, identifier) (*store.User, error)`
|
|
Finds user by ID (int) or username (string).
|
|
|
|
#### `extractImageInfo(dataURI) (type, base64, error)`
|
|
Parses data URI to extract MIME type and base64 data.
|
|
|
|
## Dependencies
|
|
|
|
### External Packages
|
|
- `github.com/labstack/echo/v4` - HTTP router and middleware
|
|
- `github.com/golang-jwt/jwt/v5` - JWT parsing and validation
|
|
- `github.com/disintegration/imaging` - Image thumbnail generation
|
|
- `golang.org/x/sync/semaphore` - Concurrency control for thumbnails
|
|
|
|
### Internal Packages
|
|
- `server/router/api/v1` - Auth constants (SessionCookieName, ClaimsMessage, etc.)
|
|
- `store` - Database operations
|
|
- `internal/profile` - Server configuration
|
|
- `plugin/storage/s3` - S3 storage client
|
|
|
|
## Configuration
|
|
|
|
### Constants
|
|
|
|
All auth-related constants are imported from `server/router/api/v1/auth.go`:
|
|
- `apiv1.SessionCookieName` - "user_session"
|
|
- `apiv1.SessionSlidingDuration` - 14 days
|
|
- `apiv1.KeyID` - "v1" (JWT key identifier)
|
|
- `apiv1.ClaimsMessage` - JWT claims struct
|
|
|
|
Package-specific constants:
|
|
- `ThumbnailCacheFolder` - ".thumbnail_cache"
|
|
- `thumbnailMaxSize` - 600px
|
|
- `SupportedThumbnailMimeTypes` - ["image/png", "image/jpeg"]
|
|
|
|
## Error Handling
|
|
|
|
All handlers return Echo HTTP errors with appropriate status codes:
|
|
|
|
```go
|
|
// Bad request
|
|
echo.NewHTTPError(http.StatusBadRequest, "message")
|
|
|
|
// Unauthorized (no auth)
|
|
echo.NewHTTPError(http.StatusUnauthorized, "message")
|
|
|
|
// Forbidden (auth but no permission)
|
|
echo.NewHTTPError(http.StatusForbidden, "message")
|
|
|
|
// Not found
|
|
echo.NewHTTPError(http.StatusNotFound, "message")
|
|
|
|
// Internal error
|
|
echo.NewHTTPError(http.StatusInternalServerError, "message").SetInternal(err)
|
|
```
|
|
|
|
## Security Considerations
|
|
|
|
### 1. XSS Prevention
|
|
SVG and HTML files are served as `application/octet-stream` to prevent script execution:
|
|
|
|
```go
|
|
if contentType == "image/svg+xml" ||
|
|
contentType == "text/html" ||
|
|
contentType == "application/xhtml+xml" {
|
|
contentType = "application/octet-stream"
|
|
}
|
|
```
|
|
|
|
### 2. Authentication
|
|
Private content requires valid session or JWT token.
|
|
|
|
### 3. Authorization
|
|
Memo visibility rules enforced before serving attachments.
|
|
|
|
### 4. Input Validation
|
|
- Attachment UID validated from database
|
|
- User identifier validated (ID or username)
|
|
- Range requests validated before processing
|
|
|
|
## Performance Optimizations
|
|
|
|
### 1. Thumbnail Caching
|
|
Thumbnails cached on disk to avoid regeneration:
|
|
- Cache location: `{data_dir}/.thumbnail_cache/`
|
|
- Filename: `{attachment_id}{extension}`
|
|
- Semaphore limits concurrent generation (max 3)
|
|
|
|
### 2. HTTP Range Requests
|
|
Video/audio files use `http.ServeContent()` for efficient streaming:
|
|
- Automatic range parsing
|
|
- Efficient memory usage (streaming, not loading full file)
|
|
- Safari-compatible partial content responses
|
|
|
|
### 3. Caching Headers
|
|
All responses include cache headers:
|
|
```
|
|
Cache-Control: public, max-age=3600
|
|
```
|
|
|
|
### 4. S3 External Links
|
|
S3 files served via presigned URLs (no server download).
|
|
|
|
## Testing
|
|
|
|
### Unit Tests (To Add)
|
|
See SAFARI_FIX.md for recommended test coverage.
|
|
|
|
### Manual Testing
|
|
```bash
|
|
# Test attachment
|
|
curl "http://localhost:8081/file/attachments/{uid}/file.jpg"
|
|
|
|
# Test avatar by ID
|
|
curl "http://localhost:8081/file/users/1/avatar"
|
|
|
|
# Test avatar by username
|
|
curl "http://localhost:8081/file/users/steven/avatar"
|
|
|
|
# Test range request
|
|
curl -H "Range: bytes=0-999" "http://localhost:8081/file/attachments/{uid}/video.mp4"
|
|
```
|
|
|
|
## Future Improvements
|
|
|
|
See SAFARI_FIX.md section "Future Improvements" for planned enhancements.
|
|
|
|
## Related Documentation
|
|
|
|
- [SAFARI_FIX.md](../../../SAFARI_FIX.md) - Full migration guide
|
|
- [server/router/api/v1/auth.go](../api/v1/auth.go) - Auth constants source of truth
|
|
- [RFC 7233](https://tools.ietf.org/html/rfc7233) - HTTP Range Requests spec
|