diff --git a/DOCS.md b/DOCS.md index e462a35..e73470e 100644 --- a/DOCS.md +++ b/DOCS.md @@ -32,6 +32,8 @@ This manual describes the Open Source Edition as it is shipped in this repositor ## Starting the server +For a new installation, use `Start-OSE.cmd` on Windows or `sh start-ose.sh` on Linux/macOS. These create a local virtual environment, install dependencies and run the first-time configuration. [GETTING_STARTED.md](GETTING_STARTED.md) explains every step, updating an existing instance, backups and the optional Linux service. The direct Python commands below remain supported. + The entry point is `server.py`. A normal start is: python3 -m pip install -r requirements.txt @@ -57,6 +59,9 @@ Logs are written to `logs/reximemo.log` and mirrored to standard output unless a |--------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `.gitignore` | Excludes Python caches/virtual environments, logs, SQLite WAL/SHM sidecars, common key/certificate/token files, `.env` and Finder metadata from commits. | | `server.py` | Process entry point, port selection, optional feature flags, logging and the proxy-compatible Twisted Site. | +| `server_config.py`, `manage.py` | Validated JSON settings, first-run administrator setup, installation checks and portable backups. | +| `Start-OSE.cmd`, `Start-OSE.ps1`, `start-ose.sh` | Windows and Linux/macOS launchers using a local virtual environment. | +| `web_features.py`, `web_ui.py` | Search, pagination, creator controls, administration forms and shared web helpers. | | `hatena.py` | Top-level HTTP resource router. It separates web traffic from DSi traffic, serves static CSS/images, loads the `hatenadir` resource tree and applies IP bans before either surface is reached. | | `reximemo.py` | Shared DSi helpers: regional URLs, current IP/account lookup, DSi keyboard values, DSi dialog/forward headers, UGO menu construction, pagination, PPM first-frame rendering and mini-Flipnote NPF conversion. | | `webapp.py` | The normal browser client. It renders Home, Browse, About, optional Docs, Flipnote, Creator's Room, account and admin pages and exposes thumbnails, inline PPM playback media and counted PPM downloads. | @@ -112,19 +117,27 @@ The DSi region root recognises `v2-xx`, `v2-eu`, `v2-us` and `v2-jp`. All four c ## Web client -The browser client is server-rendered HTML in `webapp.py`. It has no JavaScript framework and no browser session cookie. Each request asks the database for the account currently associated with the source IP. +The browser client is server-rendered HTML in `webapp.py`, `web_features.py` and `web_ui.py`. It has no JavaScript framework or account-session cookie: each request asks the database for the account currently associated with the source IP. A separate host-only CSRF cookie protects web forms; it does not change that account model. POST forms must carry its matching token. + +### Browsing and creator tools + +`/browse` shows 24 Flipnotes per page and preserves the search, sort and channel filters when moving between pages. `/search` searches Flipnotes or Creator's Rooms. `/channels` lists all channels and links to their filtered browsing pages. Creator's Rooms use the same 24-item pagination, and watch-page comments use 25 comments per page. + +An owner or administrator can open `/watch//manage`. The form changes the website title, description and channel; removal soft-deletes the Flipnote. It never rewrites the original animation. Ownership is checked on both GET and POST, using the existing OSE account/IP ownership rules. Original commenters can remove their comments; another user's account cannot acquire that right from a matching FSID alone. + +The administrator page has horizontal tabs for **Flipnotes**, **Comments**, **Channels**, **Server** and **IP bans**. Flipnote/comment lists are searchable and paginated. Their visibility controls hide or restore content with an optional reason. Hidden content remains available to administrators on the website, while normal visitors and DSi file/list routes treat it as absent. Channel removal requires moving its Flipnotes to another existing channel. General remains the fallback channel. Server settings include branding, uploaded logo, registration, guest posting and maximum PPM size. > **Login notice.** The web client uses the same deliberately simple IP identity model as the rest of OSE. Production session/device management was left out so that live authentication design, keys and account-security machinery are not copied into a public release. Shared addresses, proxies and changing IPs therefore have consequences described in the Accounts section below. | Route | Method | Behaviour | |------------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------| | `/` | GET | Home page with the most recent Flipnotes. | -| `/browse` | GET | Browse up to 100 Flipnotes. `?sort=new`, `?sort=popular` and `?sort=top` map to posted time, views/stars and stars/views orderings. | +| `/browse` | GET | Paginated Flipnotes, searchable and filterable by channel; `sort=new`, `popular` or `top` preserves the existing ordering meanings. | | `/about` | GET | Open-source credits and acknowledgements only. | | `/docs` | GET | This manual, only when documentation is enabled at server start. | | `/watch/` | GET | Flipnote detail page. It increments the view count, loads comments and shows other notes from the same Creator's Room. | | `/watch//comment` | POST | Adds a text comment using the current account or guest-IP Creator's Room. | -| `/thumb/.png` | GET | Decodes the first PPM frame and returns a cached PNG thumbnail. | +| `/thumb/.png` | GET | Renders the saved thumbnail frame with layer visibility; `?first=1` requests the player fallback's first frame. | | `/comment-thumb/.png` | GET | Returns a first-frame PNG preview for a mini-Flipnote comment. | | `/media/.ppm` | GET | Returns the original PPM inline for browser playback. It does not increment the download counter. | | `/flipnote/.ppm` | GET | Increments downloads and returns the original PPM as an attachment. | @@ -133,7 +146,7 @@ The browser client is server-rendered HTML in `webapp.py`. It has no JavaScript | `/register` | GET/POST | Creates an account and immediately associates the source IP with it. | | `/logout` | GET | Deletes the current source IP from `ip_logins`. | | `/account` | GET | Shows the signed-in username, current IP and Creator's Room link when one exists. | -| `/admin` | GET | Minimal administrator view for IP bans and Flipnote deletion. | +| `/admin` | GET | Administrator tabs for Flipnotes, comments, channels, server settings and IP bans. | | `/admin/ban` | POST | Adds or updates an IP ban and clears that IP's login row. | | `/admin/unban` | POST | Removes an IP ban. | | `/admin/delete` | POST | Soft-deletes a Flipnote record by setting `deleted=1`. | @@ -146,13 +159,13 @@ The watch page has browser playback as well as the server-generated first-frame The player starts paused. Once flipnote.js reports that the note is ready, OSE hides the first-frame image and enables the large play button and the control row. The controls provide play/pause, a seek range, elapsed/total time, sound mute/unmute and loop on/off. Seeking uses flipnote.js's `startSeek()`, `seek()` and `endSeek()` calls so playback can resume cleanly after dragging. Clicking the rendered Flipnote toggles playback after the player has been started once. -`/media/.ppm` is deliberately separate from `/flipnote/.ppm`. The media route returns the same on-disk PPM with `Content-Disposition: inline` and a one-hour public cache header, but it does not call `Database.AddDownload()`. The download route remains the explicit **Download PPM** action, increments the download counter and returns the file as an attachment. Opening or replaying a watch page therefore does not count as a file download. +`/media/.ppm` is deliberately separate from `/flipnote/.ppm`. The media route returns the same on-disk PPM with `Content-Disposition: inline`, but it does not call `Database.AddDownload()`. The download route remains the explicit **Download PPM** action, increments the download counter and returns the file as an attachment. Media and preview responses use `private, no-store`, so a previously viewed file cannot remain publicly cacheable after moderation hides it. Opening or replaying a watch page does not count as a file download. The `` first-frame preview remains in the page before JavaScript runs. If the flipnote.js global is missing, the browser cannot construct its canvas/WebAudio player, the PPM cannot be parsed, or playback reports an error, `flipnote-player.js` leaves that preview visible, disables the player controls and shows a short playback-unavailable message. The rest of the watch page still works normally. This also gives the page a useful non-JavaScript fallback. > **Browser dependency/privacy notice.** The playback library is loaded from the jsDelivr CDN rather than copied into OSE. This keeps a large third-party build out of the repository, but it means a browser visiting a watch page makes a request to jsDelivr for `flipnote.min.js`. The PPM URL itself is same-origin and is not submitted to jsDelivr by OSE. An operator who does not want that third-party browser request should self-host the pinned flipnote.js build and change the script URL in `webapp.py`. -Opening the page calls `Database.AddView()` before the Flipnote row is reloaded, so the displayed view count includes the current request. The page then loads up to 100 comments, the channel record, the Creator's Room record and a short list of other non-deleted Flipnotes from the same room. +Opening the page calls `Database.AddView()` before the Flipnote row is reloaded, so the displayed view count includes the current request. The page then loads a 25-comment page, the channel record, the Creator's Room record and a short list of other visible, non-deleted Flipnotes from the same room. The information panel displays stars, views, downloads and comment count, then the post time, channel, creator FSID, PPM filename and public identifier. These values come directly from the OSE SQLite row; there is no secondary profile, device or ownership lookup. @@ -240,7 +253,7 @@ There are no cookies or browser session identifiers in this flow. The DSi and br The DSi account page uses `username:password` entered through the Flipnote Studio keyboard. The keyboard value arrives in `X-Email-Addr`. Registration creates the account, writes the IP login, and forwards back to `sa/account.htm` using `X-DSi-Forwarder`. -The seeded administrator is `reximemo` with password `alpine`. It is created only when no account named `reximemo` exists in the opened database. +Before first-run setup, the seed administrator is `reximemo` with password `alpine`. The seed is created only when there is no administrator and setup has not been marked complete. `python manage.py setup` selects an administrator and disables the public default login when a different account is chosen. It does not remove that account's Creator's Room or media. > **Default credential notice.** The seed login exists so a fresh checkout can exercise the admin page immediately. It is public repository data, not a secret. Change or remove it before exposing an OSE instance beyond a controlled test network. @@ -272,7 +285,7 @@ The web uses the room's public ID in URLs. The DSi resource tree primarily uses > **Database/privacy notice.** OSE stores usernames, password digests, source IP login mappings, guest owner IPs and IP bans directly in SQLite. This is deliberately straightforward so the public release has no hidden account/session service or production datastore dependency. If you run a public instance, decide what IP retention, access control, backups and database-file permissions are appropriate for your environment. -Flipnote deletion is a soft delete. Listing and lookup queries require `deleted=0`; the PPM file is not erased by `delete_flipnote()`. Comments also carry a `deleted` column, although the OSE web admin does not expose comment moderation controls. +Flipnote deletion is a soft delete. Listing and lookup queries require `deleted=0`; the PPM file is not erased by `delete_flipnote()`. Flipnotes and comments also have a separate `hidden` flag and moderation reason. Public lists, counts, search, thumbnails and media lookups exclude hidden content. Administrator moderation queries explicitly request it, allowing a hidden item to be restored without deleting or re-uploading it. Sorting rules are defined in `list_flipnotes()`. Newest uses `posted_at DESC, id DESC`; Popular uses views, then stars, then posted time; Stars uses stars, then views, then posted time. @@ -302,7 +315,7 @@ PPM is the Flipnote Studio animation container handled by `Hatenatools/PPM.py`. > **Parser notice.** Hatenatools is retained primarily for compatibility with these old file formats. OSE deliberately does not wrap it in the full set of production upload validation, signature checks, anti-abuse controls and isolation measures. Treat arbitrary files from an untrusted public upload source as untrusted input and add the controls suitable for that deployment. -`ppm_first_frame_image()` asks Hatenatools to read frames but not sound, obtains frame zero, converts the returned array in Fortran order and constructs a Pillow RGBA image. Browser thumbnails convert that image to RGB and resize with nearest-neighbour resampling. +`ppm_first_frame_image()` asks Hatenatools to read frames but not sound, obtains frame zero, converts the returned array in Fortran order and constructs a Pillow RGBA image. Browser thumbnails use the PPM's saved thumbnail-frame index instead. Rendering respects the PPM hidden-layer flags, uses nearest-neighbour resampling and caches results against the source file fingerprint and requested size. Mini-comment previews still use their single frame. Rendering does not rewrite the original PPM. ### UGO / UGAR @@ -422,11 +435,11 @@ This section lists the small internal API that joins the repository together. Na ## Admin and bans -An account reaches `/admin` only when `is_admin` is 1. OSE's admin page has two jobs: maintain `ip_bans` and soft-delete Flipnotes. +An account reaches `/admin` only when `is_admin` is 1. Its tools manage channels, server customisation, hidden Flipnotes/comments, soft deletion and exact-IP bans. Content lists are paginated and searchable. Hiding and restoring use a separate state from deletion; reasons remain administrator information. Banning inserts or updates the exact text IP value and removes any login row for that address. The top-level router checks bans on every request, so a banned address cannot reach the browser pages, DSi resources or static content through the normal root resource. -Unban deletes the matching row. There are no subnet rules, expiry times, device bans, case queues, strike systems or hidden moderation states. The optional reason is limited to 200 characters and is only shown in the admin table. +Unban deletes the matching row. There are no subnet rules, expiry times, device bans, case queues or strike systems. The optional IP-ban reason is limited to 200 characters and is only shown in the admin table. Hidden content is excluded from normal DSi lookups and file responses, even when the corresponding web account is an administrator. > **Moderation notice.** IP banning and soft deletion are intentionally much simpler than RexiMemo's private moderation system. OSE leaves out device bans, behavioural/bot detection, case tooling and other production anti-abuse logic so those systems are not disclosed in a public release. Exact-IP bans are easy to understand but can affect shared networks and can be bypassed when a user's public address changes. @@ -439,7 +452,9 @@ Unban deletes the matching row. There are no subnet rules, expiry times, device | `docs:enabled` | off | Command-line flag enabling the built-in web documentation route and navigation item. | | DSi proxy | server IP, port `8080` | Set in the DSi connection settings. OSE is the Hatena HTTP proxy/server; DNS/auth/NAS must be supplied separately. | -There is no separate configuration file parser in OSE. Channel 1 and the default administrator are database seed data. DSi host names are constants in `reximemo.py`/`hatena.py`. +`server_config.py` validates `server-settings.json`. It contains the server name, tagline, welcome text, accent colour, logo, account-registration and guest-posting choices, upload limit, listen address and port. `load_config()` applies defaults; `save_config(fields)` merges recognised fields, validates the result and replaces the JSON file atomically. A bad value leaves the existing file unchanged. `REXIMEMO_CONFIG` selects another settings file and `REXIMEMO_HOST` overrides its listen address. The web settings form changes branding and posting options; address/port changes are made in setup or the file and require a restart. DSi host names remain constants in `reximemo.py`/`hatena.py`. + +`manage.py setup` asks for initial server and administrator details. `manage.py check` reports dependencies, SQLite health and configuration. `manage.py backup` writes a portable ZIP containing an online SQLite snapshot, PPM and mini-comment files, static assets, custom logo and JSON settings. Logs are excluded. The `--check` and `--backup` aliases are also accepted. Backups contain private account/IP data and should be stored privately. Stop the server before restoring an archive. > **Configuration notice.** OSE favours explicit, visible defaults over a production secrets/configuration stack. Do not place private keys, service tokens or live production credentials into the repository to compensate for missing configuration features. Keep deployment secrets outside the checkout and change the public seed administrator if the instance is reachable by others. @@ -463,6 +478,8 @@ The test suite uses the standard library `unittest` runner: `test_release_shape.py` scans the checkout for blocked key/certificate/font extensions, checks that named production-only components are absent, requires exactly three sample PPMs, and checks the OSE web branding and login wording. +The feature tests also cover upgrades to existing databases, search and page boundaries, ownership checks, channel reassignment, hiding/restoration, DSi menus and thumbnail frame/layer handling. `test_deployment.py` tests rejected configuration values, atomic updates, first-run administrator replacement and a restorable SQLite/media/settings backup. + These tests are smoke/shape checks, not a substitute for testing on a physical DSi. The DSi resource tests verify routing and generated binary structures inside Python; they do not emulate the full Flipnote Studio browser/network stack. > **Test-scope notice.** Passing this suite does not constitute a security review. In particular it does not prove resistance to hostile traffic, credential attacks, parser abuse, proxy misconfiguration or IP-identity edge cases. The release tests concentrate on keeping OSE small, reproducible, free of private material and compatible with the retained DSi paths. @@ -485,7 +502,7 @@ Extend the `CREATE TABLE IF NOT EXISTS` schema and provide an upgrade path for e ### Adding a channel -Insert a new row in `channels`. The DSi `ch.py` resource accepts any numeric channel ID present in that table. OSE's supplied home menus still post to channel 1 unless their generated URLs are changed. +Use **Admin → Channels** to add or edit a channel. Removing one moves its Flipnotes to an explicitly selected existing channel; General cannot be removed. The DSi channel directory picks up database changes on its next request, and `ch.py` accepts any existing numeric channel ID. Home-menu quick posting still defaults to channel 1. ### Changing sample data @@ -499,4 +516,4 @@ OSE is not the production RexiMemo service. The public repository does not conta The remaining code should be read with that boundary in mind. An OSE Creator's Room is an account-or-IP ownership record; a login is an IP-to-account database row; a ban is an exact IP row; browser playback is a client-side flipnote.js player with a frame-zero fallback; and the server accepts the old proxy-style DSi HTTP path rather than reproducing the removed network/authentication stack. -For public deployment work, the places most likely to need replacement or expansion are the password hashing, IP login identity, proxy/reverse-proxy IP handling, transport/authentication boundary, request-rate controls, database migrations, moderation controls, content limits/backups and operational logging. Those are outside the scope of this repository rather than hidden behind the Docs flag. +For public deployment work, the reference password hashing, IP login identity, proxy/reverse-proxy IP handling, transport/authentication boundary, request-rate controls and operational logging still need review. This update supplies basic database upgrades, moderation visibility controls, configurable upload limits and backups; it does not replace the deliberately omitted production authentication and anti-abuse systems.