Compare commits
3 Commits
main
...
multimodal
| Author | SHA1 | Date |
|---|---|---|
Jörn Fischer
|
1fcfe039a1 | |
|
Jörn Fischer
|
b37d004cb7 | |
|
Jörn Fischer
|
b0c3d32e3f |
104
README.md
104
README.md
|
|
@ -1,8 +1,8 @@
|
|||
# jcode
|
||||
|
||||
A minimal, single-file coding agent (< 1000 lines of Python) for local LLMs — with optional cloud fallback.
|
||||
A minimal, single-file coding agent (~1,200 lines of Python) for local LLMs — with optional cloud fallback.
|
||||
|
||||
jcode connects to a local **LM Studio** or **Ollama** server (or, via `settings.json`, to **OpenAI**, **Anthropic**, or **Gemini**), takes a task in plain language, and works autonomously inside a sandboxed `./code` directory: it reads, writes, and edits files, runs commands, writes its own smoke tests, and keeps fixing the code until the tests pass.
|
||||
jcode connects to a local **LM Studio** or **Ollama** server (or, via `settings.json`, to **OpenAI**, **Anthropic**, or **Gemini**), takes a task in plain language, and works autonomously inside a sandboxed `./code` directory: it reads, writes, and edits files, runs commands, searches the web, looks at images/screenshots/PDF pages, writes its own smoke tests, and keeps fixing the code until the tests pass.
|
||||
|
||||
```
|
||||
_ _
|
||||
|
|
@ -15,15 +15,25 @@ jcode connects to a local **LM Studio** or **Ollama** server (or, via `settings.
|
|||
|
||||
## Features
|
||||
|
||||
- **Autonomous test-fix loop** — the agent writes a test for every change, runs it, analyzes failures, fixes the code, and repeats until green (up to `MAX_STEPS`, default 40).
|
||||
- **Sandboxed** — all file operations are confined to `./code`; path-escape attempts (`../`, absolute paths) are blocked. Destructive shell commands (`rm -rf /`, `sudo`, `dd`, `mkfs`, shutdown, fork bombs, `curl | sh`, …) are refused.
|
||||
- **Autonomous test-fix loop** — the agent writes a test for every change, runs it, analyzes failures, fixes the code, and repeats until green (up to `MAX_STEPS`, default 200).
|
||||
- **Sandboxed** — the file tools read and write only inside `./code`; the `skills/` folder next to it is readable but read-only. Every path is resolved first (`..`, `~` and symlinks included) and then checked, so `../settings.json`, `/etc/passwd` or a symlink pointing outside are refused with a clear error, and an absolute path into `./code` is used as is instead of being re-rooted. `search_files` rejects globs with `..` and skips symlinks that leave the project. `run_command` is not path-checked (a shell can reach anything your user can); it starts in `./code`. Destructive shell commands (`rm -rf /`, `sudo`, `dd`, `mkfs`, shutdown, fork bombs, `curl | sh`, …) are refused.
|
||||
- **Zero-config local backends** — auto-detects LM Studio (port 1234) first, then Ollama (port 11434).
|
||||
- **Cloud providers via `settings.json`** — switch to OpenAI, Anthropic, or Gemini at runtime with `/backend <name>`; all are used through their OpenAI-compatible endpoints, so the single `openai` client dependency covers everything.
|
||||
- **Project guidelines via `AGENTS.md`** — project-specific rules (coding style, per-language test conventions) are loaded from `code/AGENTS.md` and injected as binding instructions.
|
||||
- **Session-wide context management** — token usage is tracked (server-reported or estimated); when the context nears its limit, older history is compacted into an LLM-written summary so long sessions keep working.
|
||||
- **Robust against flaky servers** — visible retries with backoff; if the backend goes down, the session is preserved and the task can simply be resubmitted.
|
||||
- **Transparent** — every request, tool call, and result is logged to `logFile.md`; file changes are shown as colored unified diffs in the terminal.
|
||||
- **Tools available to the model** — `list_files`, `read_file` (optionally with line numbers), `search_files` (regex grep over the sandbox, hits as `file:line: content`), `write_file`, `edit_file` (search/replace), `insert_lines`, `delete_lines`, `run_command`, `web_search` (DuckDuckGo), `fetch_url`.
|
||||
- **Skills** — reusable instruction packs in `skills/<name>/SKILL.md` (Claude-Code-compatible YAML frontmatter + Markdown, with links to further files in the skill folder). The catalog goes into the system prompt; the model loads a skill on demand with `load_skill`, or you preload one with `/<skill> task`. Five skills are included: `code-review` (diff-scoped review with verified findings), `save-load-json` (state persistence with a completeness check), `browser-test` (headless Puppeteer harness), `single-file-webapp` (patterns + static checker) and `threejs-csg` (three.js with boolean operations, loaders and exporters).
|
||||
- **Vision** — the agent can look at images, screenshots and PDF pages (see the `view_image`/`take_screenshot`/`view_pdf_page` tools below) to visually check its own work, e.g. a generated plot or a running GUI.
|
||||
- **Session-wide context management** — token usage is tracked (server-reported or estimated); when the context nears its limit, older history is compacted into an LLM-written summary so long sessions keep working. The threshold is configurable in `settings.json` (`autoCompactWindow`, Claude-Code-compatible).
|
||||
- **Adjustable reasoning effort** — defaults to `low`; `/reasoning high|medium|low|max|none` changes the model's thinking effort at runtime (backend-dependent; `none` turns it off).
|
||||
- **Robust against flaky servers** — visible retries with backoff for connection errors, timeouts and 5xx; if the backend goes down, the session is preserved and the task can simply be resubmitted. A 4xx rejection is not retried: if the request contained images, they are stripped and the task continues text-only (models without vision); if the context is too long, the history is compacted and the request repeated once; otherwise the server's message is shown.
|
||||
- **Loop guard** — an identical tool call repeated `MAX_REPEATS` times in a row first triggers a forced reflection step (expected / observed / hypotheses / one experiment); if the model loops again, the task is stopped with the session intact. A no-op `edit_file` (old = new) is rejected, so it can't feed a loop.
|
||||
- **Transparent** — every request, tool call, and result is logged to `logFile.md`; file changes are shown as colored unified diffs in the terminal. The stats line after each response shows time, generated tokens, tok/s and the prompt size the server actually received (`context N`) — compare it with the context length configured in LM Studio/Ollama: if the server's window is smaller than `autoCompactWindow`, the server silently truncates the oldest messages (including the system prompt and skills) long before jcode compacts.
|
||||
- **Tools available to the model:**
|
||||
- `list_files` (skips `node_modules`, `.git`, caches and build folders — shown as one line with their entry count — and caps the output at 300 entries), `read_file` (optionally with line numbers and a `start_line`/`end_line` range, so large files aren't re-read whole), `search_files` (regex grep over the sandbox, hits as `file:line: content`)
|
||||
- `write_file`, `edit_file` (search/replace), `insert_lines`, `delete_lines`
|
||||
- `run_command` (runs in bash wherever it is installed — on Debian/Ubuntu `/bin/sh` is dash, which lacks `pipefail`, `[[ ]]`, `source`, `{a,b}`, `echo -e` and misparses `&>`; pipes report the exit code of the first failing command; blocks known-destructive patterns; `pip`/`pip3` calls are redirected to jcode's own Python interpreter; background processes must redirect their output — `cmd > x.log 2>&1 &` — or the call blocks until the timeout; on timeout the whole process tree is killed and the output collected so far is returned)
|
||||
- `web_search` (DuckDuckGo) and `fetch_url` (fetches a page and extracts its text)
|
||||
- `view_image` (png/jpg/gif/webp/bmp/svg), `take_screenshot` (whole screen, e.g. of a GUI app under test), `view_pdf_page` (renders one PDF page) — images are delivered to the model as an inline image right after the tool result
|
||||
- `load_skill` (returns a skill's `SKILL.md` or one of its linked files)
|
||||
|
||||
## Requirements
|
||||
|
||||
|
|
@ -33,6 +43,11 @@ jcode connects to a local **LM Studio** or **Ollama** server (or, via `settings.
|
|||
- [LM Studio](https://lmstudio.ai/) with the local server running (default `http://localhost:1234`), or
|
||||
- [Ollama](https://ollama.com/) (`http://localhost:11434`), or
|
||||
- an API key for OpenAI / Anthropic / Gemini (see `settings.json` below)
|
||||
- Optional, only needed for the vision tools (the agent will tell you what's missing and how to install it when it tries to use one):
|
||||
- `Pillow` — `view_image`, `take_screenshot`, `view_pdf_page`
|
||||
- `cairosvg` — `view_image` on `.svg` files
|
||||
- `pymupdf` — `view_pdf_page`
|
||||
- `certifi` — CA bundle for `fetch_url`; falls back to the system default if not installed. Set `JCODE_INSECURE_SSL=1` to disable TLS verification (e.g. behind a re-signing corporate proxy) — use with care.
|
||||
|
||||
## Usage
|
||||
|
||||
|
|
@ -55,17 +70,23 @@ All generated files land in `./code`, which is created automatically.
|
|||
| `/models` | List models available on the current backend |
|
||||
| `/model <name>` | Switch model |
|
||||
| `/backend <name>` | Switch to `lm-studio`, `ollama`, or a provider from `settings.json` (e.g. `openai`, `anthropic`, `gemini`) |
|
||||
| `/reasoning <level>` | Set reasoning effort: `high`, `medium`, `low` (startup default), `max`, or `none` (off); no argument or `default` resets to the backend default |
|
||||
| `/max-turns N` | Set the step limit per task |
|
||||
| `Ctrl+C` | Quit |
|
||||
| `/skills` | List the skills found in `skills/` |
|
||||
| `/<skill> <task>` | Run a task with that skill's instructions preloaded (e.g. `/threejs-csg Make a plate with four holes`) |
|
||||
| `Ctrl+C` | Abort the current input or running task (session is preserved) |
|
||||
| `Ctrl+D`, or `Ctrl+C` on an empty prompt | Quit |
|
||||
|
||||
## Configuration
|
||||
|
||||
### settings.json (optional)
|
||||
|
||||
Placed next to `jcode.py`. Defines cloud providers and, optionally, the backend to start on. API keys are read from environment variables — don't put literal keys in the file.
|
||||
Placed next to `jcode.py`. Defines cloud providers, optionally the backend to start on, and the context compaction threshold. API keys are read from environment variables — don't put literal keys in the file.
|
||||
|
||||
```json
|
||||
{
|
||||
"autoCompactEnabled": true,
|
||||
"autoCompactWindow": "256k",
|
||||
"default_provider": "",
|
||||
"providers": {
|
||||
"openai": {
|
||||
|
|
@ -87,6 +108,7 @@ Placed next to `jcode.py`. Defines cloud providers and, optionally, the backend
|
|||
}
|
||||
```
|
||||
|
||||
- `autoCompactEnabled` / `autoCompactWindow` (same keys and notation as Claude Code): history compaction kicks in once the context exceeds `autoCompactWindow` tokens (`"256k"`, `"1m"` or a plain number; default 250 000). `false` disables compaction entirely. `/status` shows the active threshold.
|
||||
- `default_provider`: name of a provider to activate at startup; leave `""` to use the auto-detected local backend.
|
||||
- Each provider needs `base_url` (an OpenAI-compatible endpoint), `api_key_env` (environment variable holding the key), and optionally `model`.
|
||||
- If the file is missing or malformed, jcode prints a warning and runs purely locally.
|
||||
|
|
@ -101,16 +123,68 @@ Task> /backend anthropic
|
|||
|
||||
If `code/AGENTS.md` exists, it is loaded at session start and treated as binding project guidelines — e.g. "simplicity first", surgical changes only, and the per-language verification conventions (how to compile/test C, C++, Java, PHP, Python, HTML, CSS). Keeping language-specific rules here rather than in the system prompt keeps every request lean and lets you adapt conventions per project without touching code.
|
||||
|
||||
### Skills (optional)
|
||||
|
||||
A skill is a folder `skills/<name>/` next to `jcode.py` containing a `SKILL.md` — the same layout and frontmatter Claude Code uses, so skills can be shared between both:
|
||||
|
||||
```
|
||||
skills/
|
||||
├── browser-test/ # shot.js, actions-example.js
|
||||
├── code-review/ # review_prep.py, checklist.md
|
||||
├── save-load-json/ # js.md, python.md, check_roundtrip.py
|
||||
├── single-file-webapp/ # patterns.md, check_html.py
|
||||
└── threejs-csg/
|
||||
├── SKILL.md # frontmatter (name, description) + instructions
|
||||
├── csg.md # linked from SKILL.md as ./csg.md
|
||||
├── template.html # linked from SKILL.md as ./template.html
|
||||
└── loaders.md
|
||||
```
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: threejs-csg
|
||||
description: >
|
||||
Build 3D scenes with three.js and apply boolean operations (subtract, union,
|
||||
intersect) to meshes. Use for any three.js, WebGL or CSG task.
|
||||
---
|
||||
# Instructions ... see [csg.md](./csg.md) and [template.html](./template.html)
|
||||
```
|
||||
|
||||
- `name` and `description` are read from the frontmatter (`name` falls back to the folder name). The frontmatter parser is deliberately minimal: `key: value` lines, plus indented continuation lines for `>`/`|` block scalars — no PyYAML needed.
|
||||
- At session start, the names and descriptions of all skills are added to the system prompt, and the model can call `load_skill(name)` at any time; the body of `SKILL.md` (frontmatter stripped) is returned together with a list of the other files in the folder.
|
||||
- **Automatic selection**: at the start of every task, jcode asks the model in one tool-less request which skills match the task and injects their `SKILL.md` into the task message. Small models reliably ignore a "call load_skill first" instruction in the system prompt but reliably answer a direct question — one cheap request instead of a run that hallucinates an API. `/<skill> task` skips the selection and uses that skill.
|
||||
- Relative links in the Markdown (`./forms.md`, `[x](forms.md)`) refer to files in the skill folder; the model reads them with `load_skill(name, "forms.md")` (or `read_file`/`list_files` with `skills/<name>/<file>`, `<name>/<file>` or the absolute path — a garbled prefix in front of `/skills/<name>/` is tolerated, since models mangle long paths; the skills folder is readable but read-only: `run_command` refuses `sed -i`, redirects, `open(..., 'w')` etc. on it, so the model copies a file into the project before changing it). Scripts can be run via their absolute path, which is included in the tool result.
|
||||
- `/<skill> task` injects the skill text directly into the task so no extra tool call is needed; `/skills` lists what was found.
|
||||
|
||||
Included skills:
|
||||
|
||||
| Skill | Contents | Why |
|
||||
|---|---|---|
|
||||
| `browser-test` | `shot.js`: a Puppeteer harness that opens an HTML file via `file://` (no server), records JS exceptions / `console.error` / failed requests, runs an `actions.js` (click, shift-click, set input, key, eval, **upload** through the real file dialog, **download** through the real export button) and writes numbered screenshots; exit code 1 on any error. Probes WebGL and retries with SwiftShader flags (`--check` shows what works), traces every step and has its own watchdog, so a hang is reported with the step it stuck in instead of a silent timeout. Rules: test the control the user uses, not a stand-in; keep tests, fixtures and `node_modules` in the project. Needs `npm install puppeteer` in the project (or `CHROME_PATH` + `puppeteer-core`). | Without it the agent spends dozens of steps starting web servers and re-inventing a screenshot script, and finds runtime errors late. |
|
||||
| `single-file-webapp` | Rules for one-file ES-module apps (no inline `onclick` with module scripts, one state object + `render()`, JSON serialization with nested groups, Blob download / FileReader import, `window.__app` test hook), `patterns.md` with snippets, `check_html.py` (syntax-checks every script block with `node --check`, validates the import map, flags inline handlers and missing element ids). | Targets the most common bug class in generated single-file apps: module scope vs. inline handlers, and exporters that silently miss nested data. |
|
||||
| `code-review` | `review_prep.py` (stdlib only): scopes the review to the git diff (or given files), prints per file the changed line ranges, an outline of functions **with line ranges**, compiler/linter output restricted to bug-relevant rules (ruff/pyflakes, `node --check` incl. `<script>` blocks in HTML, gcc/clang `-Wall -Wextra`, `php -l`, `bash -n`/shellcheck) and high-signal hints; says `NOT CHECKED` when no tool is installed. `checklist.md` with questions per category and language. SKILL.md: understand intent → read one function at a time → every finding needs a concrete failure scenario, CONFIRMED by a tiny repro where cheap → ranked `REVIEW.md`, at most ~10 findings. | Small models produce long lists of plausible-sounding guesses and linter echoes. Forcing a scenario + repro per finding turns the review into something the author can act on. |
|
||||
| `save-load-json` | Save/load of complete application state as JSON, for browser apps (download/upload) and Python/Node programs (atomic file write). The rule: serialize the whole state and subtract an explicit `RUNTIME` list, so a forgotten field is saved, not lost; one constructor path for add/paste/load; `version` + `migrate`; `assertPlain` (JS) / `allow_nan=False` (Python) turn a mesh, cycle or NaN in the data into an error at save time. `check_roundtrip.py` compares a saved file with the live state's field paths and lists what was lost. Snippets in `js.md` and `python.md`, all verified. | Three Tinkercad runs produced JSON without coordinates or nested groups because the serializer listed fields by hand. |
|
||||
| `threejs-csg` | `template.html` (working scene: cube minus cylinder, import map for three / three-mesh-bvh / three-bvh-csg), `csg.md` (Brush, `updateMatrixWorld()`, operations, pitfalls), `loaders.md` (STL export/import, SVG extrusion, solid text, group = union of solids minus holes). | Pinned, verified CDN versions and the non-obvious rules (exporters want plain meshes with world matrices, not the scene or raw brushes). |
|
||||
|
||||
Skills are plain folders — add your own next to them.
|
||||
|
||||
### Constants in jcode.py
|
||||
|
||||
| Constant | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `MODEL` | (auto) | Model name; auto-corrected to the first available model |
|
||||
| `MAX_STEPS` | 40 | Tool-use steps per task |
|
||||
| `CMD_TIMEOUT` | 60 s | Timeout per shell command |
|
||||
| `MODEL` | (auto) | Model name; auto-corrected to the closest available model |
|
||||
| `MAX_STEPS` | 200 | Tool-use steps per task |
|
||||
| `MAX_REPEATS` | 3 | Identical tool calls in a row before the reflection step / stop |
|
||||
| `CMD_TIMEOUT` | 60 s | Default timeout per shell command (overridable per call via the `timeout` argument) |
|
||||
| `CONTEXT_LIMIT` | 256 000 | Model context window (tokens) |
|
||||
| `COMPACT_THRESHOLD` | 250 000 | History compaction kicks in above this |
|
||||
| `REQUEST_TIMEOUT` | 600 s | Timeout per LLM request (local models are slow) |
|
||||
| `COMPACT_THRESHOLD` | 250 000 | History compaction kicks in above this; overridden by `autoCompactWindow` in `settings.json` |
|
||||
| `AUTO_COMPACT` | `True` | Whether compaction happens at all; overridden by `autoCompactEnabled` in `settings.json` |
|
||||
| `KEEP_RECENT` | 8 | Most recent messages kept uncompressed when compacting |
|
||||
| `REQUEST_TIMEOUT` | 11 600 s | Timeout per LLM request (local models are slow) |
|
||||
| `REASONING_EFFORT` | `"low"` | Reasoning effort sent to the backend; change at runtime with `/reasoning` (`/reasoning default` = backend default) |
|
||||
| `MAX_IMAGE_SIDE` | 1280 px | Longest edge images are scaled to before being sent to the model |
|
||||
| `MAX_IMAGES_IN_CONTEXT` | 4 | Older images beyond this are replaced by a text stub to save context |
|
||||
| `SKILLS_DIR` | `skills/` next to `jcode.py` | Folder scanned for `<name>/SKILL.md` |
|
||||
|
||||
## How it works
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,643 @@
|
|||
#!/usr/bin/env python3
|
||||
# MIT License - Copyright (c) 2026 Prof. Dr. Joern Fischer (full text: see README.md)
|
||||
"""jprompt - a small, dependency-free multiline prompt editor for the terminal.
|
||||
|
||||
The editor mimics the input box of Claude Code: the text starts on its own
|
||||
line (below an optional header such as "Task>"), wraps at the terminal
|
||||
width, scrolls vertically when it is taller than the terminal and keeps the
|
||||
cursor visible at all times. The terminal is read character by character
|
||||
(cbreak-like raw mode); nothing is echoed by the terminal itself.
|
||||
|
||||
Keys
|
||||
Enter submit the text
|
||||
Ctrl+J insert a new line (Shift+Enter / Alt+Enter work too
|
||||
where the terminal reports them)
|
||||
Ctrl+C abort the input -> raises Abort (a KeyboardInterrupt
|
||||
whose .text holds the discarded input, "" if it was empty)
|
||||
Ctrl+D exit the application -> raises EOFError
|
||||
Ctrl+V / Cmd+V paste. Terminals normally deliver a paste themselves
|
||||
(bracketed paste is enabled so newlines in the pasted
|
||||
text do NOT submit). If the terminal passes Ctrl+V
|
||||
through, the system clipboard is read as a fallback.
|
||||
Arrows move the cursor. Up on the first row and Down on the
|
||||
last row do nothing.
|
||||
Home/End, Ctrl+A/Ctrl+E, Backspace, Delete, Tab (= 4 spaces)
|
||||
|
||||
Usage
|
||||
from jprompt import prompt
|
||||
text = prompt("Task>") # header is printed, editing starts below
|
||||
|
||||
Architecture (every part can be tested without a terminal)
|
||||
Buffer lines + cursor and all editing operations
|
||||
Layout wraps the Buffer lines to a width, maps cursor <-> screen cell
|
||||
Editor Buffer + Layout + rendering (ANSI) into any file-like object
|
||||
KeyReader turns raw characters into KeyEvents (escape sequences, paste)
|
||||
prompt() wires everything to the real terminal (raw mode, SIGWINCH)
|
||||
|
||||
Known limitation: terminals that *reflow* old text on resize (iTerm2,
|
||||
kitty, Windows Terminal) may leave a stale copy of the editor above the
|
||||
freshly drawn one after a resize; the editor itself and its cursor stay
|
||||
correct because every redraw is relative to the cursor position.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import codecs
|
||||
import os
|
||||
import select
|
||||
import shutil
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import unicodedata
|
||||
from dataclasses import dataclass
|
||||
|
||||
try:
|
||||
import termios
|
||||
except ImportError: # Windows
|
||||
termios = None
|
||||
try:
|
||||
import msvcrt
|
||||
except ImportError:
|
||||
msvcrt = None
|
||||
|
||||
__all__ = ["prompt", "Abort", "Buffer", "Layout", "Editor", "KeyReader", "KeyEvent", "RESIZE"]
|
||||
|
||||
class Abort(KeyboardInterrupt):
|
||||
"""Raised on Ctrl+C; `.text` is the input that was being edited."""
|
||||
|
||||
def __init__(self, text: str = "") -> None:
|
||||
super().__init__("input aborted")
|
||||
self.text = text
|
||||
|
||||
# ------------------------------------------------------------- Text helpers
|
||||
def char_width(ch: str) -> int:
|
||||
"""Display width of one character (0 for combining marks, 2 for CJK)."""
|
||||
if unicodedata.combining(ch) or unicodedata.category(ch) in ("Mn", "Me", "Cf"):
|
||||
return 0
|
||||
return 2 if unicodedata.east_asian_width(ch) in ("W", "F") else 1
|
||||
|
||||
def str_width(s: str) -> int:
|
||||
return sum(char_width(ch) for ch in s)
|
||||
|
||||
def _clean(text: str) -> str:
|
||||
"""Normalises newlines, expands tabs and drops control characters."""
|
||||
text = text.replace("\r\n", "\n").replace("\r", "\n").replace("\t", " ")
|
||||
return "".join(ch for ch in text if ch == "\n" or unicodedata.category(ch) != "Cc")
|
||||
|
||||
# ------------------------------------------------------------------- Buffer
|
||||
class Buffer:
|
||||
"""Editable text: a list of logical lines plus a cursor (row, col)."""
|
||||
|
||||
def __init__(self, text: str = "") -> None:
|
||||
self.lines: list[str] = _clean(text).split("\n")
|
||||
self.row = len(self.lines) - 1
|
||||
self.col = len(self.lines[-1])
|
||||
|
||||
# -- state
|
||||
@property
|
||||
def text(self) -> str:
|
||||
return "\n".join(self.lines)
|
||||
|
||||
@property
|
||||
def cursor(self) -> tuple[int, int]:
|
||||
return self.row, self.col
|
||||
|
||||
# -- editing
|
||||
def insert(self, text: str) -> None:
|
||||
parts = _clean(text).split("\n")
|
||||
line = self.lines[self.row]
|
||||
head, tail = line[: self.col], line[self.col :]
|
||||
if len(parts) == 1:
|
||||
self.lines[self.row] = head + parts[0] + tail
|
||||
self.col += len(parts[0])
|
||||
return
|
||||
self.lines[self.row : self.row + 1] = [head + parts[0], *parts[1:-1], parts[-1] + tail]
|
||||
self.row += len(parts) - 1
|
||||
self.col = len(parts[-1])
|
||||
|
||||
def backspace(self) -> None:
|
||||
if self.col > 0:
|
||||
line = self.lines[self.row]
|
||||
self.lines[self.row] = line[: self.col - 1] + line[self.col :]
|
||||
self.col -= 1
|
||||
elif self.row > 0:
|
||||
self.col = len(self.lines[self.row - 1])
|
||||
self.lines[self.row - 1] += self.lines.pop(self.row)
|
||||
self.row -= 1
|
||||
|
||||
def delete(self) -> None:
|
||||
line = self.lines[self.row]
|
||||
if self.col < len(line):
|
||||
self.lines[self.row] = line[: self.col] + line[self.col + 1 :]
|
||||
elif self.row + 1 < len(self.lines):
|
||||
self.lines[self.row] += self.lines.pop(self.row + 1)
|
||||
|
||||
# -- horizontal movement
|
||||
def left(self) -> None:
|
||||
if self.col > 0:
|
||||
self.col -= 1
|
||||
elif self.row > 0:
|
||||
self.row -= 1
|
||||
self.col = len(self.lines[self.row])
|
||||
|
||||
def right(self) -> None:
|
||||
if self.col < len(self.lines[self.row]):
|
||||
self.col += 1
|
||||
elif self.row + 1 < len(self.lines):
|
||||
self.row += 1
|
||||
self.col = 0
|
||||
|
||||
def home(self) -> None:
|
||||
self.col = 0
|
||||
|
||||
def end(self) -> None:
|
||||
self.col = len(self.lines[self.row])
|
||||
|
||||
# ------------------------------------------------------------------- Layout
|
||||
@dataclass(frozen=True)
|
||||
class VisualRow:
|
||||
line: int # index of the logical line
|
||||
start: int # first character (inclusive)
|
||||
end: int # last character (exclusive)
|
||||
last: bool # last visual row of its logical line?
|
||||
|
||||
class Layout:
|
||||
"""Wraps logical lines to `width` display columns.
|
||||
|
||||
A logical line of exactly `width` columns is NOT split: the cursor may sit
|
||||
at display column `width` (which is why Editor uses terminal width - 1).
|
||||
A cursor at the end of a non-last visual row is shown at the start of the
|
||||
next row - the same convention prompt_toolkit uses.
|
||||
"""
|
||||
|
||||
def __init__(self, lines: list[str], width: int) -> None:
|
||||
self.lines = lines
|
||||
self.width = max(1, width)
|
||||
self.rows: list[VisualRow] = []
|
||||
for li, line in enumerate(lines):
|
||||
start = acc = 0
|
||||
for i, ch in enumerate(line):
|
||||
w = char_width(ch)
|
||||
if acc + w > self.width and i > start:
|
||||
self.rows.append(VisualRow(li, start, i, False))
|
||||
start, acc = i, 0
|
||||
acc += w
|
||||
self.rows.append(VisualRow(li, start, len(line), True))
|
||||
|
||||
def row_text(self, r: VisualRow) -> str:
|
||||
return self.lines[r.line][r.start : r.end]
|
||||
|
||||
def cursor_to_visual(self, row: int, col: int) -> tuple[int, int]:
|
||||
"""(logical row, col) -> (visual row index, display column)."""
|
||||
for vr, r in enumerate(self.rows):
|
||||
if r.line == row and (r.start <= col < r.end or (r.last and col == r.end)):
|
||||
return vr, str_width(self.lines[row][r.start : col])
|
||||
return len(self.rows) - 1, 0 # unreachable for a consistent buffer
|
||||
|
||||
def visual_to_cursor(self, vr: int, want_col: int) -> tuple[int, int]:
|
||||
"""(visual row index, wanted display column) -> nearest (row, col)."""
|
||||
r = self.rows[vr]
|
||||
line = self.lines[r.line]
|
||||
limit = r.end if r.last else r.end - 1 # end of a wrapped row belongs to the next row
|
||||
col, acc = r.start, 0
|
||||
while col < limit:
|
||||
w = char_width(line[col])
|
||||
if acc + w > want_col:
|
||||
break
|
||||
acc, col = acc + w, col + 1
|
||||
return r.line, col
|
||||
|
||||
# ---------------------------------------------------------------- KeyEvents
|
||||
@dataclass(frozen=True)
|
||||
class KeyEvent:
|
||||
kind: str # text, newline, enter, backspace, delete, left, right, up,
|
||||
# down, home, end, ctrl_c, ctrl_d, resize, ignore
|
||||
text: str = "" # only for kind == "text" (typed char or pasted block)
|
||||
|
||||
RESIZE = object() # sentinel a key source returns when the window was resized
|
||||
|
||||
# ------------------------------------------------------------------- Editor
|
||||
class Editor:
|
||||
"""Buffer + rendering. `out` needs write()/flush(); `size()` -> (cols, rows).
|
||||
|
||||
The editor assumes the terminal cursor is at column 0 of the line where
|
||||
the text should start when render() is first called. All later drawing is
|
||||
relative to the cursor, so scrolling of the terminal does not matter.
|
||||
"""
|
||||
|
||||
def __init__(self, out, size, text: str = "") -> None:
|
||||
self.buf = Buffer(text)
|
||||
self.out = out
|
||||
self._size = size
|
||||
self._want_col: int | None = None # sticky column for Up/Down
|
||||
self._top = 0 # first visible visual row (vertical scrolling)
|
||||
self._cur_row = 0 # screen row of the cursor relative to the first drawn row
|
||||
self._drawn = 0 # number of rows drawn by the last render()
|
||||
self.hidden_above = 0 # visual rows scrolled out of view (shown as markers)
|
||||
self.hidden_below = 0
|
||||
|
||||
# -- geometry
|
||||
@property
|
||||
def width(self) -> int:
|
||||
"""Wrap width: one less than the terminal so the cursor never touches the auto-wrap column."""
|
||||
return max(1, self._size()[0] - 1)
|
||||
|
||||
@property
|
||||
def height(self) -> int:
|
||||
"""Max. visible rows: one less than the terminal so a header line above survives."""
|
||||
return max(1, self._size()[1] - 1)
|
||||
|
||||
@property
|
||||
def text(self) -> str:
|
||||
return self.buf.text
|
||||
|
||||
def layout(self) -> Layout:
|
||||
return Layout(self.buf.lines, self.width)
|
||||
|
||||
# -- keys
|
||||
def handle(self, key: KeyEvent) -> bool:
|
||||
"""Applies one key; returns True when the text was submitted."""
|
||||
k = key.kind
|
||||
if k == "ctrl_c":
|
||||
raise Abort(self.text)
|
||||
if k == "ctrl_d":
|
||||
raise EOFError
|
||||
if k == "enter":
|
||||
return True
|
||||
if k == "text":
|
||||
self.buf.insert(key.text)
|
||||
elif k == "newline":
|
||||
self.buf.insert("\n")
|
||||
elif k == "backspace":
|
||||
self.buf.backspace()
|
||||
elif k == "delete":
|
||||
self.buf.delete()
|
||||
elif k == "left":
|
||||
self.buf.left()
|
||||
elif k == "right":
|
||||
self.buf.right()
|
||||
elif k == "home":
|
||||
self.buf.home()
|
||||
elif k == "end":
|
||||
self.buf.end()
|
||||
elif k == "up":
|
||||
self.move_up()
|
||||
elif k == "down":
|
||||
self.move_down()
|
||||
if k not in ("up", "down", "ignore"):
|
||||
self._want_col = None
|
||||
self.render()
|
||||
return False
|
||||
|
||||
def move_up(self) -> bool:
|
||||
return self._vertical(-1)
|
||||
|
||||
def move_down(self) -> bool:
|
||||
return self._vertical(+1)
|
||||
|
||||
def _vertical(self, delta: int) -> bool:
|
||||
"""Moves one *visual* row; returns False at the top/bottom edge."""
|
||||
lay = self.layout()
|
||||
vr, vc = lay.cursor_to_visual(*self.buf.cursor)
|
||||
target = vr + delta
|
||||
if not 0 <= target < len(lay.rows):
|
||||
return False
|
||||
if self._want_col is None:
|
||||
self._want_col = vc
|
||||
self.buf.row, self.buf.col = lay.visual_to_cursor(target, self._want_col)
|
||||
return True
|
||||
|
||||
# -- drawing
|
||||
def _viewport(self, vr: int, n: int) -> tuple[int, int, bool, bool]:
|
||||
"""Chooses the first visible row so the cursor row is on screen.
|
||||
|
||||
Returns (top, rows shown, marker above?, marker below?). Each marker
|
||||
line ("... N more lines ...") costs one row of the available height,
|
||||
so the choice is iterated until it is stable.
|
||||
"""
|
||||
h, top = self.height, self._top
|
||||
above = below = False
|
||||
for _ in range(3):
|
||||
avail = max(1, h - above - below)
|
||||
if vr < top:
|
||||
top = vr
|
||||
elif vr >= top + avail:
|
||||
top = vr - avail + 1
|
||||
top = max(0, min(top, n - avail))
|
||||
if h < 3: # no room for markers
|
||||
return top, avail, False, False
|
||||
new = (top > 0, top + avail < n)
|
||||
if new == (above, below):
|
||||
return top, avail, above, below
|
||||
above, below = new
|
||||
return top, avail, above, below
|
||||
|
||||
def _marker(self, count: int, where: str) -> str:
|
||||
text = f"\u2026 {count} more line{'s' if count != 1 else ''} {where} \u2026"
|
||||
if str_width(text) > self.width:
|
||||
arrow = "\u2191" if where == "above" else "\u2193"
|
||||
text = f"\u2026{count}{arrow}"
|
||||
return f"\x1b[2m{text}\x1b[0m"
|
||||
|
||||
def render(self) -> None:
|
||||
lay = self.layout()
|
||||
vr, vc = lay.cursor_to_visual(*self.buf.cursor)
|
||||
n = len(lay.rows)
|
||||
top, avail, above, below = self._viewport(vr, n)
|
||||
self._top = top
|
||||
self.hidden_above, self.hidden_below = top, max(0, n - top - avail)
|
||||
|
||||
lines = [lay.row_text(r) for r in lay.rows[top : top + avail]]
|
||||
if above:
|
||||
lines.insert(0, self._marker(self.hidden_above, "above"))
|
||||
if below:
|
||||
lines.append(self._marker(self.hidden_below, "below"))
|
||||
crow = vr - top + int(above)
|
||||
|
||||
parts = []
|
||||
if self._cur_row > 0: # back to the first drawn row
|
||||
parts.append(f"\x1b[{self._cur_row}A")
|
||||
parts.append("\r\x1b[J") # column 0, clear to end of screen
|
||||
parts.append("\r\n".join(lines))
|
||||
back = len(lines) - 1 - crow
|
||||
if back > 0:
|
||||
parts.append(f"\x1b[{back}A")
|
||||
parts.append("\r")
|
||||
if vc > 0:
|
||||
parts.append(f"\x1b[{vc}C")
|
||||
self.out.write("".join(parts))
|
||||
self.out.flush()
|
||||
self._cur_row, self._drawn = crow, len(lines)
|
||||
|
||||
def finish(self) -> None:
|
||||
"""Moves the cursor below the text and starts a fresh line."""
|
||||
down = self._drawn - 1 - self._cur_row
|
||||
self.out.write((f"\x1b[{down}B" if down > 0 else "") + "\r\n")
|
||||
self.out.flush()
|
||||
|
||||
# ---------------------------------------------------------------- KeyReader
|
||||
class KeyReader:
|
||||
"""Decodes characters from a `source` into KeyEvents.
|
||||
|
||||
source.read() blocks; returns one character or RESIZE
|
||||
source.pending(timeout) True if a character arrives within `timeout` s
|
||||
"""
|
||||
|
||||
ESC_TIMEOUT = 0.05
|
||||
|
||||
_CTRL = {"\r": "enter", "\n": "newline", "\x7f": "backspace", "\x08": "backspace",
|
||||
"\x03": "ctrl_c", "\x04": "ctrl_d", "\x01": "home", "\x05": "end"}
|
||||
_CSI_FINAL = {"A": "up", "B": "down", "C": "right", "D": "left", "H": "home", "F": "end"}
|
||||
_CSI_TILDE = {"1": "home", "7": "home", "4": "end", "8": "end", "3": "delete"}
|
||||
|
||||
def __init__(self, source) -> None:
|
||||
self.source = source
|
||||
self._resized = False
|
||||
|
||||
def read_key(self) -> KeyEvent:
|
||||
if self._resized:
|
||||
self._resized = False
|
||||
return KeyEvent("resize")
|
||||
ch = self.source.read()
|
||||
if ch is RESIZE:
|
||||
return KeyEvent("resize")
|
||||
if ch == "\x1b":
|
||||
return self._escape()
|
||||
if ch == "\x16": # Ctrl+V passed through by the terminal
|
||||
text = _clipboard()
|
||||
return KeyEvent("text", text) if text else KeyEvent("ignore")
|
||||
if ch == "\t":
|
||||
return KeyEvent("text", " ")
|
||||
if ch in self._CTRL:
|
||||
return KeyEvent(self._CTRL[ch])
|
||||
if unicodedata.category(ch) != "Cc":
|
||||
return KeyEvent("text", ch)
|
||||
return KeyEvent("ignore")
|
||||
|
||||
def _next(self) -> str:
|
||||
"""Next character inside an escape sequence; remembers resizes for later."""
|
||||
while True:
|
||||
ch = self.source.read()
|
||||
if ch is RESIZE:
|
||||
self._resized = True
|
||||
continue
|
||||
return ch
|
||||
|
||||
def _escape(self) -> KeyEvent:
|
||||
if not self.source.pending(self.ESC_TIMEOUT):
|
||||
return KeyEvent("ignore") # a lone Esc
|
||||
c = self._next()
|
||||
if c in ("\r", "\n"):
|
||||
return KeyEvent("newline") # Alt+Enter
|
||||
if c == "O": # SS3: cursor keys in application mode
|
||||
return KeyEvent(self._CSI_FINAL.get(self._next(), "ignore"))
|
||||
if c != "[":
|
||||
return KeyEvent("ignore") # Alt+<key>
|
||||
seq = ""
|
||||
while True:
|
||||
d = self._next()
|
||||
seq += d
|
||||
if "\x40" <= d <= "\x7e":
|
||||
break
|
||||
if len(seq) > 32:
|
||||
return KeyEvent("ignore")
|
||||
return self._csi(seq)
|
||||
|
||||
def _csi(self, seq: str) -> KeyEvent:
|
||||
final, params = seq[-1], seq[:-1]
|
||||
if seq == "200~": # bracketed paste
|
||||
return self._paste()
|
||||
if final in self._CSI_FINAL:
|
||||
return KeyEvent(self._CSI_FINAL[final])
|
||||
if final == "~":
|
||||
return KeyEvent(self._CSI_TILDE.get(params.split(";")[0], "ignore"))
|
||||
if final == "u" and params.split(";")[0] == "13": # kitty protocol: Shift/Ctrl+Enter
|
||||
return KeyEvent("newline")
|
||||
return KeyEvent("ignore")
|
||||
|
||||
def _paste(self) -> KeyEvent:
|
||||
end, buf = "\x1b[201~", []
|
||||
while True:
|
||||
buf.append(self._next())
|
||||
if len(buf) >= len(end) and "".join(buf[-len(end):]) == end:
|
||||
return KeyEvent("text", "".join(buf[: -len(end)]))
|
||||
|
||||
def _clipboard() -> str:
|
||||
"""Best-effort read of the system clipboard (fallback for a raw Ctrl+V)."""
|
||||
for cmd in (["pbpaste"], ["wl-paste", "-n"], ["xclip", "-selection", "clipboard", "-o"],
|
||||
["xsel", "-b", "-o"], ["powershell", "-NoProfile", "-Command", "Get-Clipboard"]):
|
||||
if shutil.which(cmd[0]):
|
||||
try:
|
||||
return subprocess.run(cmd, capture_output=True, text=True, timeout=2).stdout
|
||||
except Exception: # noqa: BLE001
|
||||
continue
|
||||
return ""
|
||||
|
||||
# -------------------------------------------------------- Terminal plumbing
|
||||
class _TtySource:
|
||||
"""Character source for a POSIX tty; `wake_fd` (optional) signals resizes."""
|
||||
|
||||
def __init__(self, fd: int, wake_fd: int | None = None) -> None:
|
||||
self.fd, self.wake = fd, wake_fd
|
||||
self._dec = codecs.getincrementaldecoder("utf-8")(errors="replace")
|
||||
self._buf: list[str] = [] # decoded but not yet consumed characters
|
||||
|
||||
def read(self):
|
||||
fds = [self.fd] if self.wake is None else [self.fd, self.wake]
|
||||
while not self._buf:
|
||||
ready, _, _ = select.select(fds, [], [])
|
||||
if self.wake in ready:
|
||||
try:
|
||||
os.read(self.wake, 64)
|
||||
except OSError:
|
||||
pass
|
||||
return RESIZE
|
||||
# Read whole chunks: a large paste must be drained quickly, the
|
||||
# kernel's tty input queue is small (1 KB on macOS).
|
||||
b = os.read(self.fd, 65536)
|
||||
if not b:
|
||||
raise EOFError
|
||||
self._buf.extend(self._dec.decode(b))
|
||||
return self._buf.pop(0)
|
||||
|
||||
def pending(self, timeout: float) -> bool:
|
||||
if self._buf:
|
||||
return True
|
||||
ready, _, _ = select.select([self.fd], [], [], timeout)
|
||||
return bool(ready)
|
||||
|
||||
class _WinSource:
|
||||
"""Character source for the Windows console (msvcrt); special keys become CSI sequences."""
|
||||
|
||||
_SPECIAL = {"H": "A", "P": "B", "K": "D", "M": "C", "G": "H", "O": "F", "S": "3~"}
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._queue: list[str] = []
|
||||
|
||||
def read(self):
|
||||
if self._queue:
|
||||
return self._queue.pop(0)
|
||||
ch = msvcrt.getwch()
|
||||
if ch in ("\x00", "\xe0"):
|
||||
code = self._SPECIAL.get(msvcrt.getwch())
|
||||
if code is None:
|
||||
return self.read()
|
||||
self._queue.extend("[" + code)
|
||||
return "\x1b"
|
||||
return ch
|
||||
|
||||
def pending(self, timeout: float) -> bool:
|
||||
return bool(self._queue) or msvcrt.kbhit()
|
||||
|
||||
class _RawMode:
|
||||
"""Character-at-a-time input without echo; Enter stays '\\r', Ctrl+J stays '\\n'."""
|
||||
|
||||
def __init__(self, fd: int) -> None:
|
||||
self.fd = fd
|
||||
|
||||
def __enter__(self):
|
||||
if termios is None:
|
||||
return self
|
||||
self.old = termios.tcgetattr(self.fd)
|
||||
new = termios.tcgetattr(self.fd)
|
||||
new[0] &= ~(termios.ICRNL | termios.INLCR | termios.IXON) # iflag
|
||||
new[3] &= ~(termios.ECHO | termios.ICANON | termios.ISIG | termios.IEXTEN) # lflag
|
||||
new[6][termios.VMIN], new[6][termios.VTIME] = 1, 0
|
||||
termios.tcsetattr(self.fd, termios.TCSANOW, new)
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc):
|
||||
if termios is not None:
|
||||
termios.tcsetattr(self.fd, termios.TCSADRAIN, self.old)
|
||||
|
||||
class _ResizeWatcher:
|
||||
"""Turns SIGWINCH into a readable byte on `self.fd` (None when unsupported)."""
|
||||
|
||||
def __enter__(self):
|
||||
self.fd = None
|
||||
if not hasattr(signal, "SIGWINCH"):
|
||||
return self
|
||||
r, w = os.pipe()
|
||||
try:
|
||||
os.set_blocking(w, False)
|
||||
self._old_wake = signal.set_wakeup_fd(w)
|
||||
self._old_handler = signal.signal(signal.SIGWINCH, lambda *_: None)
|
||||
self.fd, self._pipe = r, (r, w)
|
||||
except (ValueError, OSError): # e.g. not in the main thread
|
||||
os.close(r), os.close(w)
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc):
|
||||
if self.fd is None:
|
||||
return
|
||||
signal.signal(signal.SIGWINCH, self._old_handler)
|
||||
signal.set_wakeup_fd(self._old_wake)
|
||||
for fd in self._pipe:
|
||||
os.close(fd)
|
||||
|
||||
def _terminal_size() -> tuple[int, int]:
|
||||
size = shutil.get_terminal_size((80, 24))
|
||||
return size.columns, size.lines
|
||||
|
||||
def _enable_windows_vt() -> None:
|
||||
try:
|
||||
import ctypes
|
||||
k32 = ctypes.windll.kernel32
|
||||
handle, mode = k32.GetStdHandle(-11), ctypes.c_uint32()
|
||||
if k32.GetConsoleMode(handle, ctypes.byref(mode)):
|
||||
k32.SetConsoleMode(handle, mode.value | 0x0004) # ENABLE_VIRTUAL_TERMINAL_PROCESSING
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
def _interactive() -> bool:
|
||||
try:
|
||||
return sys.stdin.isatty() and sys.stdout.isatty() and (termios or msvcrt) is not None
|
||||
except (AttributeError, ValueError):
|
||||
return False
|
||||
|
||||
# ------------------------------------------------------------------- prompt
|
||||
def prompt(header: str = "", text: str = "") -> str:
|
||||
"""Prints `header` (if any) and edits `text` on the following line(s).
|
||||
|
||||
Returns the entered text on Enter. Raises Abort (a KeyboardInterrupt with
|
||||
the current input in `.text`) on Ctrl+C and EOFError on Ctrl+D. Without a
|
||||
terminal (pipe), one line is read from stdin instead.
|
||||
"""
|
||||
if header:
|
||||
print(header, flush=True)
|
||||
if not _interactive():
|
||||
line = sys.stdin.readline()
|
||||
if not line:
|
||||
raise EOFError
|
||||
return line.rstrip("\r\n")
|
||||
|
||||
out = sys.stdout
|
||||
editor = Editor(out, _terminal_size, text)
|
||||
if termios is None:
|
||||
_enable_windows_vt()
|
||||
source, fd = _WinSource(), None
|
||||
else:
|
||||
fd = sys.stdin.fileno()
|
||||
with _RawMode(fd), _ResizeWatcher() as watcher:
|
||||
if termios is not None:
|
||||
source = _TtySource(fd, watcher.fd)
|
||||
reader = KeyReader(source)
|
||||
out.write("\x1b[?25h\x1b[?2004h") # cursor visible, bracketed paste on
|
||||
try:
|
||||
editor.render()
|
||||
while not editor.handle(reader.read_key()):
|
||||
pass
|
||||
finally:
|
||||
editor.finish()
|
||||
out.write("\x1b[?2004l")
|
||||
out.flush()
|
||||
return editor.text
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
result = prompt("jprompt demo - Enter submits, Ctrl+J = new line, Ctrl+C aborts, Ctrl+D exits\nText>")
|
||||
print(f"You entered: {result!r}")
|
||||
except KeyboardInterrupt:
|
||||
print("(aborted)")
|
||||
except EOFError:
|
||||
print("(exit)")
|
||||
|
|
@ -1,4 +1,6 @@
|
|||
{
|
||||
"autoCompactEnabled": true,
|
||||
"autoCompactWindow": "256k",
|
||||
"default_provider": "",
|
||||
"providers": {
|
||||
"openai": {
|
||||
|
|
@ -15,6 +17,16 @@
|
|||
"base_url": "https://generativelanguage.googleapis.com/v1beta/openai/",
|
||||
"api_key_env": "GEMINI_API_KEY",
|
||||
"model": "gemini-2.5-pro"
|
||||
},
|
||||
"qwen": {
|
||||
"base_url": "https://api.scaleway.ai/6bfec461-302a-4568-84c6-c141c7836828/v1",
|
||||
"api_key_env": "QWEN_API_KEY",
|
||||
"model": "qwen3.8-27b"
|
||||
},
|
||||
"openrouter": {
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"api_key_env": "OPENROUTER_API_KEY",
|
||||
"model": "qwen/qwen3.8-27b:free"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -0,0 +1,120 @@
|
|||
---
|
||||
name: browser-test
|
||||
description: >
|
||||
Test and screenshot HTML/JS web apps in a headless browser (Puppeteer): catch
|
||||
JavaScript exceptions, console errors and 404s, drive the UI (click, shift-click,
|
||||
type, set inputs) and take screenshots to check visually. Use whenever a task
|
||||
produces an HTML page, canvas/WebGL app or any browser UI that must be verified.
|
||||
---
|
||||
|
||||
# Headless browser testing
|
||||
|
||||
Use the ready-made harness [shot.js](./shot.js). Do NOT write your own Puppeteer
|
||||
launcher and do NOT start an HTTP server: the page is opened via `file://`,
|
||||
CDN imports work from there.
|
||||
|
||||
## Setup (once per project)
|
||||
```
|
||||
cp <skill-dir>/shot.js . && (npm ls puppeteer > /dev/null 2>&1 || npm install puppeteer > npm.log 2>&1) && echo ok
|
||||
```
|
||||
`<skill-dir>` is the absolute skill directory shown at the top of this text.
|
||||
The skill folder is read-only - you work with the copy `./shot.js`, and if the
|
||||
harness lacks something, extend THAT copy with edit_file. `npm install puppeteer`
|
||||
downloads a matching Chrome (~150 MB, 1-2 min). With an existing Chrome:
|
||||
`CHROME_PATH=/path/to/chrome` plus `npm install puppeteer-core`.
|
||||
|
||||
## Run
|
||||
```
|
||||
node shot.js app.html # load, record errors, shots/01_initial.png
|
||||
node shot.js app.html actions.js # + drive the UI
|
||||
node shot.js --check # environment probe: which Chrome flags give WebGL
|
||||
```
|
||||
Always call it with `run_command(..., timeout=120)`: the harness has its own
|
||||
watchdog (60 s, `SHOT_TIMEOUT=90 node shot.js ...` to raise it) and must be
|
||||
allowed to report before the outer timeout kills it.
|
||||
|
||||
Exit codes: 1 = the page threw / `console.error` / failed resource / actions.js
|
||||
threw (error list at the end); 2 = setup problem (puppeteer missing, no WebGL
|
||||
with any flag set); 3 = watchdog - the output ends with `WATCHDOG ... stuck in
|
||||
step: <helper + args>`, i.e. the last traced step is the one that hangs.
|
||||
Every helper call is traced as `» name args`, so partial output always tells
|
||||
you how far the test got. Screenshots go to `./shots/NN_name.png`; look at
|
||||
them with `view_image`.
|
||||
|
||||
## Environment problems are not app bugs
|
||||
- `No WebGL context with any flag set`, `WebGL context could not be created`,
|
||||
a watchdog in step `goto`, or `Navigation timeout`: the headless browser or
|
||||
the network is the problem. Run `node shot.js --check`, try
|
||||
`CHROME_PATH=/Applications/Google Chrome.app/Contents/MacOS/Google Chrome`
|
||||
(macOS) or `npm i puppeteer@latest`. NEVER restructure the app to run without
|
||||
WebGL, wrap `renderer` in `if (renderer)`, or add "GL not supported" fallbacks
|
||||
- that degrades the product the user asked for while the test setup is the
|
||||
actual culprit. If the environment cannot be fixed, say so in the summary and
|
||||
verify with `check_html.py` plus a hook-level test of the non-3D logic.
|
||||
- A watchdog inside `eval`/`download`/`upload`: the app function called never
|
||||
returns (infinite loop, awaiting a promise that never resolves, a download
|
||||
that never starts). Read that app function; don't patch shot.js.
|
||||
- Don't edit `shot.js` to work around a failing test. If a helper is really
|
||||
missing, add it to the project copy once, with a clear name, and keep using
|
||||
the traced helpers (`h.page` gives you the full Puppeteer API anyway).
|
||||
|
||||
## actions.js
|
||||
```js
|
||||
module.exports = async (h) => {
|
||||
await h.clickEl('#add-box'); // click a button
|
||||
await h.clickEl('#add-sphere', { shift: true }); // shift-click an element
|
||||
await h.set('#pos-x', 12); // set an input, fires input+change
|
||||
await h.click(640, 400, { shift: true }); // (shift-)click at page coordinates
|
||||
await h.key('Delete');
|
||||
const n = await h.eval(() => window.__app.state.objects.length); // read app state
|
||||
if (n !== 1) throw new Error(`expected 1 object, got ${n}`);
|
||||
await h.upload('#btn-load-svg', 'fixtures/circle.svg'); // REAL import: button -> file dialog -> app code
|
||||
const f = await h.download('#btn-save-json'); // REAL export: button -> <a download> -> file
|
||||
if (!f.text.includes('"position"')) throw new Error('saved file lacks position: ' + f.text);
|
||||
await h.shot('final');
|
||||
};
|
||||
```
|
||||
Helpers: `shot`, `click`, `clickEl`, `set`, `key`, `upload`, `download`, `eval`,
|
||||
`wait`, `page` (full Puppeteer API), `errors`. Complete example:
|
||||
[actions-example.js](./actions-example.js).
|
||||
|
||||
## How to test an app without going in circles
|
||||
1. **First run without actions.** Fix every reported error before testing features.
|
||||
2. **Drive state through the test hook, not through pixels.** The app exposes
|
||||
`window.__app = { state, add, select, toJSON, ... }` (see the single-file-webapp
|
||||
skill). To test "group a solid with a hole", do
|
||||
`h.eval(() => { __app.add('box'); __app.add('cylinder'); __app.select([1, 2]); })`,
|
||||
then click the Group button and assert on `__app.state`. Canvas picking is
|
||||
tested ONCE, separately: add one object at a known position, click its
|
||||
projected screen position (`__app.project(id)` or compute it), assert selection.
|
||||
3. **Never change the app to make a test pass** (e.g. "buttons now select
|
||||
existing objects") - change the test, or fix a real bug in the app.
|
||||
4. **Two failures with the same message = stop editing the test.** Print the
|
||||
relevant state in the error (`JSON.stringify(await h.eval(...))`), then read
|
||||
the app function involved (`search_files` + `read_file` with a line range).
|
||||
5. One actions file per feature area (`test_shapes.js`, `test_group.js`,
|
||||
`test_files.js`), each under ~40 lines; short, deterministic, throws on
|
||||
unexpected state. One 200-line file that runs everything means every fix
|
||||
re-runs everything and one hang hides all other results. Add a `shot()`
|
||||
after every visible change. Keep `h.eval` results small (ids, counts,
|
||||
plain numbers) - returning meshes or whole state objects serializes
|
||||
megabytes and can stall the protocol.
|
||||
6. WebGL apps: check the screenshot for a rendered object, not just a dark canvas.
|
||||
7. **Test the path the user takes, not a stand-in for it.** A feature wired to a
|
||||
button, file dialog or download is verified ONLY by `h.clickEl`, `h.upload`
|
||||
or `h.download` on that control. Calling the underlying function via
|
||||
`__app`, or re-implementing the import inside the test with the library
|
||||
exposed on `window`, proves nothing about the button - that is exactly how
|
||||
"verified" features turn out broken for the user. Use `__app` for setup and
|
||||
assertions, the real control for the action under test. Write fixture files
|
||||
(`fixtures/box.stl`, `fixtures/circle.svg`, a saved `scene.json`) once and
|
||||
keep them.
|
||||
8. **Never delete the test setup.** `shot.js`, `test_*.js`, `fixtures/`,
|
||||
`node_modules/` and `package.json` stay in the project so the user can rerun
|
||||
`node shot.js app.html test_x.js` after you are done. Add `shots/` and
|
||||
`node_modules/` to `.gitignore` instead of removing them.
|
||||
9. Change an existing test with `edit_file`; rewriting a 5 KB test with
|
||||
`write_file` for a two-line change is slow and error-prone.
|
||||
10. The final summary names each feature with the test file and the kind of
|
||||
check (real control / hook only / screenshot only). "Hook only" is not
|
||||
"verified".
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
// Example actions for shot.js: a 3D editor with a left shape menu, a canvas
|
||||
// in the middle and a properties panel on the right. Adapt selectors/coords.
|
||||
module.exports = async (h) => {
|
||||
await h.clickEl('#add-box');
|
||||
await h.clickEl('#add-cylinder');
|
||||
await h.shot('two_shapes');
|
||||
|
||||
// select both via shift-click in the canvas, then group
|
||||
await h.click(640, 420);
|
||||
await h.click(720, 420, { shift: true });
|
||||
await h.shot('multi_selected');
|
||||
await h.clickEl('#group');
|
||||
|
||||
// verify through the app's test hook instead of guessing from pixels
|
||||
const groups = await h.eval(() => window.__app.objects.filter(o => o.type === 'group').length);
|
||||
if (groups !== 1) throw new Error(`expected 1 group, got ${groups}`);
|
||||
await h.shot('grouped');
|
||||
|
||||
// import through the REAL file dialog (button -> <input type=file> -> app code)
|
||||
await h.upload('#import-svg', 'fixtures/circle.svg');
|
||||
const types = await h.eval(() => window.__app.state.objects.map(o => o.type));
|
||||
if (!types.includes('svg')) throw new Error('SVG import via button added nothing: ' + types);
|
||||
await h.shot('svg_imported');
|
||||
|
||||
// export through the REAL button (-> <a download> -> file on disk)
|
||||
const json = await h.download('#save-json');
|
||||
const data = JSON.parse(json.text);
|
||||
if (!data.objects?.[0]?.position) throw new Error('saved JSON has no position: ' + json.text.slice(0, 200));
|
||||
const stl = await h.download('#export-stl');
|
||||
if (!stl.text.startsWith('solid')) throw new Error('STL export is not ASCII STL');
|
||||
};
|
||||
|
|
@ -0,0 +1,227 @@
|
|||
#!/usr/bin/env node
|
||||
// shot.js - headless browser harness for single-file web apps (Puppeteer).
|
||||
//
|
||||
// node shot.js <app.html> [actions.js] [outdir] run (default outdir: ./shots)
|
||||
// node shot.js --check probe the browser: WebGL, flags
|
||||
//
|
||||
// Opens the HTML via file:// (no web server needed), records every JS
|
||||
// exception, console error and failed request, runs the optional actions
|
||||
// script and writes numbered screenshots to outdir. Exit code 1 if any error
|
||||
// was recorded, 2 on setup problems, 3 on the internal watchdog (default 60 s,
|
||||
// SHOT_TIMEOUT=<seconds> to change) - so it never hangs silently.
|
||||
//
|
||||
// actions.js exports one async function: module.exports = async (h) => { ... }
|
||||
// with the helper object h (every call is traced to stdout as "» name args"):
|
||||
// h.page Puppeteer page (full API)
|
||||
// await h.shot('name') screenshot -> outdir/NN_name.png (waits 2 frames first)
|
||||
// await h.click(x, y, {shift}) mouse click at page coordinates (shift = multi-select)
|
||||
// await h.clickEl(selector, {shift}) click a DOM element (button, menu entry)
|
||||
// await h.set(selector, value) set an <input>/<select> value and dispatch 'input'+'change'
|
||||
// await h.key('Delete') press a key (Puppeteer key names)
|
||||
// await h.upload('#btn-load-svg', 'x.svg') click the button that opens the file
|
||||
// dialog and feed it the file (real import path)
|
||||
// const {name,text} = await h.download('#btn-save') click the button that
|
||||
// triggers a download, return the file (real export path)
|
||||
// await h.eval(fn, ...args) run fn in the page and return its result (keep it JSON-small)
|
||||
// await h.wait(ms)
|
||||
// h.errors errors collected so far (array of strings)
|
||||
//
|
||||
// Chrome: Puppeteer downloads its own on 'npm install puppeteer'. To use an
|
||||
// existing browser set CHROME_PATH=/path/to/chrome (then 'puppeteer-core' is enough).
|
||||
// WebGL: the harness probes WebGL at start and retries with other GPU flags
|
||||
// (SwiftShader) if the first launch has none. If every flag set fails, that is an
|
||||
// ENVIRONMENT problem - fix it with --check, never by making the app run without WebGL.
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
const CHECK = args[0] === '--check';
|
||||
const [htmlArg, actionsArg, outArg] = args;
|
||||
if (!htmlArg) { console.error('usage: node shot.js <app.html> [actions.js] [outdir] | --check'); process.exit(2); }
|
||||
|
||||
// Resolve puppeteer from the PROJECT (cwd), not from this skill folder.
|
||||
const req = require('module').createRequire(path.join(process.cwd(), 'package.json'));
|
||||
let puppeteer;
|
||||
try { puppeteer = req('puppeteer'); } catch {
|
||||
try { puppeteer = req('puppeteer-core'); } catch {
|
||||
console.error('puppeteer not found in ' + process.cwd() + ' - run: npm install puppeteer'); process.exit(2);
|
||||
}
|
||||
}
|
||||
|
||||
// GPU flag sets tried in order until a page gets a WebGL context.
|
||||
const FLAG_SETS = [
|
||||
['--no-sandbox', '--enable-unsafe-swiftshader', '--ignore-gpu-blocklist'],
|
||||
['--no-sandbox', '--enable-unsafe-swiftshader', '--use-angle=swiftshader', '--use-gl=angle'],
|
||||
['--no-sandbox', '--enable-unsafe-swiftshader', '--use-gl=swiftshader', '--disable-gpu'],
|
||||
['--no-sandbox', '--enable-unsafe-swiftshader', '--use-angle=swiftshader-webgl'],
|
||||
];
|
||||
const CACHE = path.join(process.cwd(), '.shot-flags.json');
|
||||
|
||||
function die(e) { console.error(e.stack || e); process.exit(2); }
|
||||
|
||||
async function launch(flags) {
|
||||
return puppeteer.launch({
|
||||
headless: true,
|
||||
executablePath: process.env.CHROME_PATH || undefined,
|
||||
args: flags,
|
||||
protocolTimeout: 30000,
|
||||
});
|
||||
}
|
||||
|
||||
async function hasWebGL(browser) {
|
||||
const p = await browser.newPage();
|
||||
try {
|
||||
await p.setContent('<canvas id=c></canvas>');
|
||||
return await p.evaluate(() => {
|
||||
const c = document.getElementById('c');
|
||||
const gl = c.getContext('webgl2') || c.getContext('webgl');
|
||||
if (!gl) return null;
|
||||
const d = gl.getExtension('WEBGL_debug_renderer_info');
|
||||
return d ? gl.getParameter(d.UNMASKED_RENDERER_WEBGL) : 'unknown renderer';
|
||||
});
|
||||
} finally { await p.close(); }
|
||||
}
|
||||
|
||||
// Launch with the cached flag set, else probe the sets; returns {browser, flags, renderer}.
|
||||
async function launchWithWebGL() {
|
||||
let order = FLAG_SETS;
|
||||
try { const c = JSON.parse(fs.readFileSync(CACHE, 'utf8')); order = [c, ...FLAG_SETS.filter(f => f.join() !== c.join())]; } catch {}
|
||||
let last = null;
|
||||
for (const flags of order) {
|
||||
const browser = await launch(flags);
|
||||
const renderer = await hasWebGL(browser).catch(() => null);
|
||||
if (renderer) { fs.writeFileSync(CACHE, JSON.stringify(flags)); return { browser, flags, renderer }; }
|
||||
last = flags; await browser.close();
|
||||
}
|
||||
console.error('No WebGL context with any flag set (last tried: ' + last.join(' ') + ').\n' +
|
||||
'This is an environment problem (headless Chrome / GPU), NOT an app bug: do not change the app.\n' +
|
||||
'Try: CHROME_PATH=<path to a system Chrome>, or update puppeteer (npm i puppeteer@latest), then node shot.js --check');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
async function check() {
|
||||
for (const flags of FLAG_SETS) {
|
||||
let browser; try { browser = await launch(flags); } catch (e) { console.log('launch failed:', flags.join(' '), '-', e.message); continue; }
|
||||
const r = await hasWebGL(browser).catch(e => 'ERR ' + e.message);
|
||||
console.log((r && !String(r).startsWith('ERR') ? 'WebGL OK ' : 'no WebGL ') + flags.join(' ') + (r ? ' -> ' + r : ''));
|
||||
await browser.close();
|
||||
}
|
||||
console.log('chrome:', process.env.CHROME_PATH || (puppeteer.executablePath ? puppeteer.executablePath() : '(bundled)'));
|
||||
}
|
||||
|
||||
if (CHECK) { check().catch(die); } else (async () => {
|
||||
const html = path.resolve(htmlArg);
|
||||
const outdir = path.resolve(outArg || 'shots');
|
||||
fs.mkdirSync(outdir, { recursive: true });
|
||||
const errors = [];
|
||||
let step = 'launch';
|
||||
|
||||
// Watchdog: never hang silently. Prints where it was and what it collected.
|
||||
const budget = (parseInt(process.env.SHOT_TIMEOUT, 10) || 60) * 1000;
|
||||
const watchdog = setTimeout(() => {
|
||||
console.log(`\nWATCHDOG: no progress after ${budget / 1000}s - stuck in step: ${step}`);
|
||||
if (errors.length) { console.log(`${errors.length} error(s) so far:`); errors.forEach(e => console.log(' ' + e)); }
|
||||
process.exit(3);
|
||||
}, budget);
|
||||
|
||||
const { browser, flags, renderer } = await launchWithWebGL();
|
||||
console.log(`chrome flags: ${flags.join(' ')} | WebGL: ${renderer}`);
|
||||
const page = await browser.newPage();
|
||||
await page.setViewport({ width: 1280, height: 800 });
|
||||
page.setDefaultTimeout(10000);
|
||||
|
||||
page.on('pageerror', e => errors.push(`pageerror: ${e.message}`));
|
||||
page.on('console', m => { if (m.type() === 'error') errors.push(`console.error: ${m.text()}`); });
|
||||
page.on('requestfailed', r => errors.push(`request failed: ${r.url()} (${r.failure()?.errorText})`));
|
||||
page.on('response', r => { if (r.status() >= 400) errors.push(`HTTP ${r.status()}: ${r.url()}`); });
|
||||
|
||||
let n = 0;
|
||||
const frames = () => page.evaluate(() => new Promise(r => requestAnimationFrame(() => requestAnimationFrame(r))));
|
||||
const trace = (name, ...a) => { step = `${name} ${a.map(x => typeof x === 'function' ? '[fn]' : JSON.stringify(x)).join(' ')}`.slice(0, 120); console.log('» ' + step); };
|
||||
const h = {
|
||||
page, errors,
|
||||
wait: ms => new Promise(r => setTimeout(r, ms)),
|
||||
eval: (fn, ...a) => { trace('eval', fn, ...a); return page.evaluate(fn, ...a); },
|
||||
async shot(name) {
|
||||
trace('shot', name);
|
||||
await frames();
|
||||
const file = path.join(outdir, `${String(++n).padStart(2, '0')}_${name}.png`);
|
||||
await page.screenshot({ path: file });
|
||||
console.log(`shot ${path.relative(process.cwd(), file)}`);
|
||||
},
|
||||
async click(x, y, { shift = false } = {}) {
|
||||
trace('click', x, y, shift ? '+shift' : '');
|
||||
if (shift) await page.keyboard.down('Shift');
|
||||
await page.mouse.click(x, y);
|
||||
if (shift) await page.keyboard.up('Shift');
|
||||
await frames();
|
||||
},
|
||||
async clickEl(selector, { shift = false } = {}) {
|
||||
trace('clickEl', selector, shift ? '+shift' : '');
|
||||
if (shift) await page.keyboard.down('Shift');
|
||||
await page.click(selector);
|
||||
if (shift) await page.keyboard.up('Shift');
|
||||
await frames();
|
||||
},
|
||||
async set(selector, value) {
|
||||
trace('set', selector, value);
|
||||
await page.$eval(selector, (el, v) => {
|
||||
el.value = v;
|
||||
el.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
el.dispatchEvent(new Event('change', { bubbles: true }));
|
||||
}, String(value));
|
||||
await frames();
|
||||
},
|
||||
async key(k) { trace('key', k); await page.keyboard.press(k); await frames(); },
|
||||
// Real file import: click the button that opens the file dialog, feed it a file.
|
||||
async upload(selector, filePath) {
|
||||
trace('upload', selector, filePath);
|
||||
const [chooser] = await Promise.all([page.waitForFileChooser({ timeout: 5000 }), page.click(selector)]);
|
||||
await chooser.accept([path.resolve(filePath)]);
|
||||
await frames(); await h.wait(300);
|
||||
},
|
||||
// Real file export: click the button that triggers <a download>, return {name, path, text}.
|
||||
async download(selector, { timeout = 5000 } = {}) {
|
||||
trace('download', selector);
|
||||
const dir = path.join(outdir, 'downloads');
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
for (const f of fs.readdirSync(dir)) fs.unlinkSync(path.join(dir, f));
|
||||
const cdp = await page.createCDPSession();
|
||||
await cdp.send('Browser.setDownloadBehavior', { behavior: 'allow', downloadPath: dir });
|
||||
await page.click(selector);
|
||||
const t0 = Date.now();
|
||||
while (Date.now() - t0 < timeout) {
|
||||
const done = fs.readdirSync(dir).filter(f => !f.endsWith('.crdownload'));
|
||||
if (done.length) {
|
||||
const file = path.join(dir, done[0]);
|
||||
return { name: done[0], path: file, text: fs.readFileSync(file, 'utf8') };
|
||||
}
|
||||
await h.wait(100);
|
||||
}
|
||||
throw new Error(`download: nothing arrived within ${timeout} ms after clicking ${selector}`);
|
||||
},
|
||||
};
|
||||
|
||||
step = 'goto (loading page + CDN modules)';
|
||||
await page.goto('file://' + html, { waitUntil: 'load', timeout: 45000 });
|
||||
// give module scripts / CDN imports a moment; wait for the test hook if the app has one
|
||||
await page.waitForFunction(() => window.__app !== undefined, { timeout: 15000 }).catch(() => {});
|
||||
await frames();
|
||||
await h.shot('initial');
|
||||
|
||||
if (actionsArg) {
|
||||
try { await require(path.resolve(actionsArg))(h); }
|
||||
catch (e) { errors.push(`actions (in step "${step}"): ${e.stack || e}`); }
|
||||
}
|
||||
step = 'closing browser';
|
||||
await browser.close();
|
||||
clearTimeout(watchdog);
|
||||
|
||||
if (errors.length) {
|
||||
console.log(`\n${errors.length} error(s):`);
|
||||
errors.forEach(e => console.log(' ' + e));
|
||||
process.exit(1);
|
||||
}
|
||||
console.log('\nno errors');
|
||||
})().catch(die);
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
---
|
||||
name: code-review
|
||||
description: >
|
||||
Review existing code for bugs, security problems and risky design: scope the
|
||||
review (git diff or files), outline and lint with a helper script, read
|
||||
function by function, verify each suspected bug with a concrete failing input,
|
||||
and write a ranked REVIEW.md. Use when the task asks to review, audit, check,
|
||||
or find bugs/problems in code - not for writing new features.
|
||||
---
|
||||
|
||||
# Code review
|
||||
|
||||
The goal is a short list of REAL problems the author would want to fix, each
|
||||
with evidence. Ten verified findings beat fifty guesses. A review does not
|
||||
change the code unless the task explicitly asks for fixes.
|
||||
|
||||
## 1. Scope and groundwork (one command)
|
||||
```
|
||||
python <skill-dir>/review_prep.py # git: changed files; else all source files
|
||||
python <skill-dir>/review_prep.py a.py b.js # only these files
|
||||
python <skill-dir>/review_prep.py --base main # diff against a branch/commit
|
||||
```
|
||||
`<skill-dir>` is the absolute directory at the top of this text. Per file you
|
||||
get: changed line ranges, an outline (`start-end def name`), compiler/linter
|
||||
output and hints. Linter output and hints are LEADS, not findings - many are
|
||||
harmless in context. `NOT CHECKED` means no compiler was available: say so in
|
||||
the report.
|
||||
|
||||
## 2. Understand intent before judging
|
||||
Read what the code is supposed to do: the task text, README/AGENTS.md, module
|
||||
docstrings, the tests. A "bug" is a difference between intent and behaviour -
|
||||
without knowing the intent you can only report style.
|
||||
|
||||
## 3. Read function by function
|
||||
- Use the outline: `read_file(path, start_line=a, end_line=b)` for ONE function
|
||||
at a time. Never read a whole large file; never re-read a range you have.
|
||||
- Order: changed ranges first, then the functions they call and the callers
|
||||
of changed functions (`search_files("name\\(")`), then hotspots: input
|
||||
parsing, file/network I/O, error handling, concurrency, anything that builds
|
||||
shell commands, SQL, HTML or paths from input.
|
||||
- For each function ask the questions in [checklist.md](./checklist.md).
|
||||
Load it once at the start: `load_skill("code-review", "checklist.md")`.
|
||||
|
||||
## 4. Verify every finding
|
||||
For each suspected problem, write down a concrete failure scenario:
|
||||
**input/state -> what happens -> what should happen**. Then:
|
||||
- If it is cheap, PROVE it: a 1-5 line repro with `run_command`
|
||||
(`python -c "from calc import average; average([])"`, a tiny test file,
|
||||
`node -e ...`). Mark it **CONFIRMED** and quote the output.
|
||||
- If a repro is not practical (needs network, GUI, a race), mark it
|
||||
**PLAUSIBLE** and state the exact preconditions.
|
||||
- If you cannot state a concrete scenario, it is not a finding - drop it or
|
||||
list it under "Questions for the author".
|
||||
Never put repro files into the project's source tree: use a `_review/` folder
|
||||
and mention it in the report.
|
||||
|
||||
## 5. Report: write REVIEW.md
|
||||
```markdown
|
||||
# Review: <scope> (files, base, date)
|
||||
Checked with: <linters/compilers that ran; what was NOT CHECKED>
|
||||
|
||||
## Findings (most severe first)
|
||||
### 1. [HIGH] <one-line claim> - `file.py:42` - CONFIRMED
|
||||
**Scenario:** `average([])` -> ZeroDivisionError is swallowed, returns None;
|
||||
callers format None as "None %" (report.py:88).
|
||||
**Evidence:** `python -c ...` -> `None`
|
||||
**Fix:** raise ValueError for empty input (or return 0.0 if that is the intent).
|
||||
|
||||
## Questions for the author
|
||||
- ...
|
||||
|
||||
## Not reviewed
|
||||
- vendor/, generated files, ...
|
||||
```
|
||||
Severity: **HIGH** = wrong results, data loss, crash on normal input, security
|
||||
hole. **MEDIUM** = crash/misbehaviour on edge cases, resource leaks, silent
|
||||
error swallowing. **LOW** = maintainability problems that will likely cause
|
||||
bugs later. Pure style (naming, formatting, import order) only if asked.
|
||||
|
||||
At most ~10 findings; if there are more, keep the most severe and say how many
|
||||
smaller ones exist. End your answer with the count per severity and the path
|
||||
to REVIEW.md - do not paste the whole report into the answer.
|
||||
|
||||
## Pitfalls
|
||||
- Reporting a linter line verbatim without reading the code around it.
|
||||
- "Could be None" without showing a call path where it IS None.
|
||||
- Suggesting a rewrite instead of the smallest fix.
|
||||
- Flagging framework conventions as bugs (read how the codebase does it elsewhere).
|
||||
- Running out of steps on a big codebase: after the changed ranges and
|
||||
hotspots, stop and list the rest under "Not reviewed".
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
# Review checklist
|
||||
|
||||
Ask these per function. Skip what does not apply; stop at the first real problem
|
||||
in a function and verify it before going on.
|
||||
|
||||
## Inputs and edge cases
|
||||
- Empty (`[]`, `""`, `{}`), zero, negative, very large, None/null/undefined?
|
||||
- First/last element, off-by-one in ranges and slices, inclusive vs exclusive end?
|
||||
- Unicode, whitespace, different line endings, case differences?
|
||||
- Duplicate keys/ids, unsorted input, input that changes while iterating?
|
||||
|
||||
## Errors
|
||||
- Which calls can fail (I/O, parsing, network, lookups)? What happens then?
|
||||
- Errors swallowed (`except: pass`, empty `catch {}`), or turned into a wrong
|
||||
default that flows on (None, 0, "")?
|
||||
- Is the error message useful? Is cleanup done on the error path
|
||||
(files/sockets/locks closed - `with`, `try/finally`, RAII)?
|
||||
- Retries without limit or backoff? Timeouts missing on network/subprocess calls?
|
||||
|
||||
## State and data flow
|
||||
- Is every variable initialised on every path? Used after being freed/closed?
|
||||
- Shared mutable state: module globals, class attributes, mutable default
|
||||
arguments, objects passed by reference and then modified?
|
||||
- Two representations of the same thing that can drift apart (cache vs source,
|
||||
UI vs model, serialized format vs in-memory)?
|
||||
- Save/load and export/import: is every field written AND read back?
|
||||
|
||||
## Security (wherever input crosses a boundary)
|
||||
- Shell: command built from input with string formatting / `shell=True`?
|
||||
- Paths: input joined into a path without checking `..` and absolute paths?
|
||||
- SQL / HTML / JS: string concatenation instead of parameters / escaping
|
||||
(`innerHTML`, template strings into the DOM)?
|
||||
- Secrets in code, logs or error messages? `eval`/`exec`/`pickle` on input?
|
||||
- Downloads/requests: TLS verification disabled, URLs from input (SSRF)?
|
||||
|
||||
## Concurrency and async
|
||||
- Awaited everything that returns a promise/coroutine? Fire-and-forget errors lost?
|
||||
- Check-then-act races (exists() then open(), read-modify-write without lock)?
|
||||
- Event handlers registered repeatedly (leak) or never removed?
|
||||
|
||||
## Performance (only if it matters at the expected size)
|
||||
- Work inside a loop that belongs outside (re-reading files, re-compiling regexes,
|
||||
O(n^2) membership tests on lists)?
|
||||
- Unbounded growth: caches, histories, listeners, logs?
|
||||
|
||||
## Language specifics
|
||||
- **Python:** `is` vs `==`, late-binding closures in loops, `except Exception`
|
||||
catching too much, integer vs float division, iterator consumed twice,
|
||||
`open()` without `encoding=`.
|
||||
- **JavaScript:** `==` coercion, `this` in callbacks, missing `await`,
|
||||
`parseInt` without radix, `forEach` with async callbacks, module scope vs
|
||||
inline `onclick`, floating point money/coordinates.
|
||||
- **C/C++:** buffer sizes (`strcpy`, `sprintf`, missing `\0`), signed/unsigned
|
||||
comparisons, integer overflow, missing `return`, ownership/double free,
|
||||
uninitialised memory, `sizeof(pointer)` vs `sizeof(array)`.
|
||||
- **Java:** `==` on strings/boxed types, unclosed streams (try-with-resources),
|
||||
`equals` without `hashCode`, swallowed `InterruptedException`.
|
||||
- **PHP:** `==` juggling, unescaped output (`htmlspecialchars`), SQL without
|
||||
prepared statements, `include` with input.
|
||||
- **Shell:** unquoted variables, missing `set -euo pipefail`, `cd` without check.
|
||||
|
||||
## Tests
|
||||
- Do tests exist for the changed behaviour? Do they assert the result, or only
|
||||
that nothing crashed? Would they fail if the bug above were present?
|
||||
|
|
@ -0,0 +1,243 @@
|
|||
#!/usr/bin/env python3
|
||||
"""review_prep.py - mechanical groundwork for a code review (stdlib only).
|
||||
|
||||
python review_prep.py changed files (git) or all source files
|
||||
python review_prep.py src/a.py b.js only these files
|
||||
python review_prep.py --base main git diff against a branch/commit
|
||||
|
||||
Prints, per file: language, size, the changed line ranges (git), an outline of
|
||||
functions/classes WITH line ranges (so you can read_file exactly one function),
|
||||
compiler/linter results, and a few high-signal hints. Hints and linter output
|
||||
are LEADS to check, not findings.
|
||||
"""
|
||||
import ast, os, re, shutil, subprocess, sys, tempfile
|
||||
from pathlib import Path
|
||||
|
||||
SRC = {".py": "python", ".js": "js", ".mjs": "js", ".cjs": "js", ".ts": "ts", ".html": "html",
|
||||
".htm": "html", ".c": "c", ".h": "c", ".cpp": "cpp", ".cc": "cpp", ".hpp": "cpp",
|
||||
".java": "java", ".php": "php", ".sh": "sh", ".go": "go", ".rs": "rust"}
|
||||
SKIP_DIRS = {".git", "node_modules", "__pycache__", ".venv", "venv", "dist", "build", "shots", ".idea"}
|
||||
MAX_LINES = 40 # per section, keeps the report small
|
||||
|
||||
|
||||
def run(cmd, **kw):
|
||||
try:
|
||||
p = subprocess.run(cmd, capture_output=True, text=True, timeout=60, **kw)
|
||||
return p.returncode, (p.stdout + p.stderr).strip()
|
||||
except (OSError, subprocess.TimeoutExpired) as e:
|
||||
return -1, str(e)
|
||||
|
||||
|
||||
def cap(lines, n=MAX_LINES):
|
||||
lines = [l for l in lines if l.strip()]
|
||||
return lines[:n] + ([f"... {len(lines) - n} more"] if len(lines) > n else [])
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ scope
|
||||
def git_root():
|
||||
rc, out = run(["git", "rev-parse", "--show-toplevel"])
|
||||
return Path(out) if rc == 0 else None
|
||||
|
||||
|
||||
def changed_files(base):
|
||||
rc, out = run(["git", "diff", "--name-only", base] if base else ["git", "diff", "--name-only", "HEAD"])
|
||||
files = out.splitlines() if rc == 0 else []
|
||||
rc, out = run(["git", "ls-files", "--others", "--exclude-standard"])
|
||||
files += out.splitlines() if rc == 0 else []
|
||||
root = git_root()
|
||||
return [Path(os.path.relpath(root / f)) for f in dict.fromkeys(files)
|
||||
if (root / f).is_file() and (root / f).suffix in SRC]
|
||||
|
||||
|
||||
def changed_ranges(path, base):
|
||||
"""Line ranges of added/modified lines in the NEW file, from the unified diff."""
|
||||
rc, out = run(["git", "diff", "-U0", base or "HEAD", "--", str(path)])
|
||||
if rc != 0 or not out:
|
||||
return None # untracked or no git: whole file is "new"
|
||||
rngs = []
|
||||
for m in re.finditer(r"^@@ -\S+ \+(\d+)(?:,(\d+))? @@", out, re.M):
|
||||
start, n = int(m.group(1)), int(m.group(2) or 1)
|
||||
if n:
|
||||
rngs.append(f"{start}-{start + n - 1}" if n > 1 else str(start))
|
||||
return rngs
|
||||
|
||||
|
||||
def all_files():
|
||||
out = []
|
||||
for dp, dns, fns in os.walk("."):
|
||||
dns[:] = [d for d in dns if d not in SKIP_DIRS and not d.startswith(".")]
|
||||
out += [Path(dp) / f for f in fns if Path(f).suffix in SRC]
|
||||
return sorted(out)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- outline
|
||||
def outline_python(text):
|
||||
try:
|
||||
tree = ast.parse(text)
|
||||
except SyntaxError as e:
|
||||
return [f"SyntaxError line {e.lineno}: {e.msg}"], []
|
||||
rows, hints = [], []
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
|
||||
kind = "class" if isinstance(node, ast.ClassDef) else "def"
|
||||
rows.append((node.lineno, f"{node.lineno}-{node.end_lineno} {kind} {node.name}"))
|
||||
if kind == "def":
|
||||
for d in node.args.defaults + node.args.kw_defaults:
|
||||
if isinstance(d, (ast.List, ast.Dict, ast.Set)):
|
||||
hints.append(f"{d.lineno}: mutable default argument in {node.name}()")
|
||||
if isinstance(node, ast.ExceptHandler):
|
||||
if node.type is None:
|
||||
hints.append(f"{node.lineno}: bare 'except:' (also catches KeyboardInterrupt/SystemExit)")
|
||||
if len(node.body) == 1 and isinstance(node.body[0], ast.Pass):
|
||||
hints.append(f"{node.lineno}: exception silently swallowed (except ...: pass)")
|
||||
if isinstance(node, ast.Call):
|
||||
f = node.func
|
||||
name = f.id if isinstance(f, ast.Name) else f.attr if isinstance(f, ast.Attribute) else ""
|
||||
if name in ("eval", "exec"):
|
||||
hints.append(f"{node.lineno}: {name}() - check where the input comes from")
|
||||
if any(k.arg == "shell" and isinstance(k.value, ast.Constant) and k.value.value for k in node.keywords):
|
||||
hints.append(f"{node.lineno}: subprocess with shell=True - check quoting of interpolated values")
|
||||
if isinstance(f, ast.Name) and f.id == "open" and not any(k.arg == "encoding" for k in node.keywords) and \
|
||||
not (len(node.args) > 1 and isinstance(node.args[1], ast.Constant) and "b" in str(node.args[1].value)):
|
||||
hints.append(f"{node.lineno}: open() without encoding= (platform-dependent on Windows)")
|
||||
if isinstance(node, ast.Compare) and any(isinstance(o, (ast.Eq, ast.NotEq)) for o in node.ops) and \
|
||||
any(isinstance(c, ast.Constant) and c.value is None for c in node.comparators):
|
||||
hints.append(f"{node.lineno}: '== None' - use 'is None'")
|
||||
return [r for _, r in sorted(rows)], sorted(set(hints), key=lambda h: int(h.split(":")[0]))
|
||||
|
||||
|
||||
JS_DEF = re.compile(r"^\s*(?:export\s+)?(?:async\s+)?(?:function\s*\*?\s*(\w+)|class\s+(\w+)|"
|
||||
r"(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?(?:function|\([^)]*\)\s*=>|\w+\s*=>))")
|
||||
C_DEF = re.compile(r"^(?!\s*(?:if|for|while|switch|return|else)\b)[\w:<>,\*\s&]+?\b(\w+)\s*\([^;]*\)\s*(?:const)?\s*\{?\s*$")
|
||||
|
||||
|
||||
def outline_regex(text, lang):
|
||||
rows, hints = [], []
|
||||
lines = text.splitlines()
|
||||
for i, line in enumerate(lines, 1):
|
||||
if lang in ("js", "ts", "html"):
|
||||
m = JS_DEF.match(line)
|
||||
if m:
|
||||
rows.append(f"{i} {next(g for g in m.groups() if g)}")
|
||||
if re.search(r"\son\w+\s*=\s*[\"']", line) and lang == "html":
|
||||
hints.append(f"{i}: inline event handler - with <script type=module> it cannot reach module scope")
|
||||
if re.search(r"\binnerHTML\s*=(?!=)", line):
|
||||
hints.append(f"{i}: innerHTML assignment - XSS if the value contains user input")
|
||||
if re.search(r"[^=!]==[^=]", line) and not re.search(r"['\"].*==.*['\"]", line):
|
||||
hints.append(f"{i}: '==' (loose equality)")
|
||||
elif lang in ("c", "cpp", "java") and C_DEF.match(line) and not line.startswith((" ", "\t")):
|
||||
rows.append(f"{i} {C_DEF.match(line).group(1)}")
|
||||
if lang in ("c", "cpp") and re.search(r"\b(gets|strcpy|strcat|sprintf)\s*\(", line):
|
||||
hints.append(f"{i}: unbounded {re.search(r'(gets|strcpy|strcat|sprintf)', line).group(1)}()")
|
||||
if re.search(r"\b(TODO|FIXME|XXX|HACK)\b", line):
|
||||
hints.append(f"{i}: {line.strip()[:90]}")
|
||||
return rows, hints
|
||||
|
||||
|
||||
# ---------------------------------------------------------- tools/linters
|
||||
def node_msg(msg):
|
||||
"""node --check output without stack frames: 'file:line', the source line, the error."""
|
||||
keep = [l for l in msg.splitlines() if l.strip() and not l.lstrip().startswith("at ")
|
||||
and not l.startswith("Node.js v")]
|
||||
return keep[:4]
|
||||
|
||||
|
||||
def html_scripts_check(path, text):
|
||||
if not shutil.which("node"):
|
||||
return []
|
||||
out = []
|
||||
for i, (attrs, body) in enumerate(re.findall(r"<script\b([^>]*)>(.*?)</script>", text, re.S | re.I), 1):
|
||||
if "src=" in attrs or "importmap" in attrs or not body.strip():
|
||||
continue
|
||||
line0 = text[:text.find(body)].count("\n")
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".mjs" if "module" in attrs else ".js",
|
||||
delete=False, encoding="utf-8") as f:
|
||||
f.write(body)
|
||||
rc, msg = run(["node", "--check", f.name])
|
||||
os.unlink(f.name)
|
||||
if rc:
|
||||
m = re.search(r":(\d+)\n", msg)
|
||||
where = f"line ~{line0 + int(m.group(1))}" if m else f"script {i}"
|
||||
out.append(f"node --check {where}: " + " | ".join(node_msg(msg.replace(f.name, "<script>"))))
|
||||
return out
|
||||
|
||||
|
||||
TOOLS = {"python": [sys.executable], "js": ["node"], "html": ["node"], "c": ["gcc", "clang"],
|
||||
"cpp": ["g++", "clang++", "clang"], "php": ["php"], "sh": ["bash"]}
|
||||
|
||||
|
||||
def linters(path, lang, text):
|
||||
"""List of messages, or None if no checker for this language is installed."""
|
||||
if not any(shutil.which(x) for x in TOOLS.get(lang, [])):
|
||||
return None
|
||||
p, out = str(path), []
|
||||
if lang == "python":
|
||||
if shutil.which("ruff"):
|
||||
# --isolated: ignore local style configs; only rules that indicate bugs
|
||||
out += run(["ruff", "check", "--isolated", "--quiet", "--output-format=concise",
|
||||
"--select", "E9,F,B,PLE,PLW0120,RUF006", "--ignore", "F541", p])[1].splitlines()
|
||||
else:
|
||||
rc, msg = run([sys.executable, "-m", "pyflakes", p])
|
||||
if "No module named pyflakes" not in msg:
|
||||
out += msg.splitlines()
|
||||
rc, msg = run([sys.executable, "-m", "py_compile", p])
|
||||
out += msg.splitlines() if rc else []
|
||||
elif lang == "js" and shutil.which("node"):
|
||||
rc, msg = run(["node", "--check", p]); out += node_msg(msg) if rc else []
|
||||
elif lang == "html":
|
||||
out += html_scripts_check(path, text)
|
||||
elif lang in ("c", "cpp"):
|
||||
cc = next(filter(None, map(shutil.which, TOOLS[lang])), None)
|
||||
if cc:
|
||||
# real compile (not -fsyntax-only): flow warnings like missing return / uninitialized need it
|
||||
rc, msg = run([cc, "-O1", "-Wall", "-Wextra", "-c", "-o", os.devnull, p]); out += msg.splitlines()
|
||||
elif lang == "php" and shutil.which("php"):
|
||||
rc, msg = run(["php", "-l", p]); out += msg.splitlines() if rc else []
|
||||
elif lang == "sh" and shutil.which("bash"):
|
||||
rc, msg = run(["bash", "-n", p]); out += msg.splitlines()
|
||||
if shutil.which("shellcheck"):
|
||||
out += run(["shellcheck", "-f", "gcc", p])[1].splitlines()
|
||||
for name in (os.path.abspath(p), p, os.path.relpath(p)): # longest first
|
||||
out = [l.replace(name + ":", "line ") for l in out]
|
||||
return list(dict.fromkeys(out))
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- main
|
||||
def main(argv):
|
||||
base = None
|
||||
if "--base" in argv:
|
||||
i = argv.index("--base"); base = argv[i + 1]; del argv[i:i + 2]
|
||||
in_git = git_root() is not None
|
||||
if argv:
|
||||
files, mode = [Path(a) for a in argv], "given files"
|
||||
elif in_git and (files := changed_files(base)):
|
||||
mode = f"changed vs {base or 'HEAD'} (git)"
|
||||
else:
|
||||
files, mode = all_files(), "all source files"
|
||||
print(f"# Review prep - {len(files)} file(s), scope: {mode}\n")
|
||||
for path in files:
|
||||
if not path.is_file():
|
||||
print(f"## {path}\n(not found)\n"); continue
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
lang = SRC.get(path.suffix, "?")
|
||||
rows, hints = outline_python(text) if lang == "python" else outline_regex(text, lang)
|
||||
print(f"## {path} ({lang}, {text.count(chr(10)) + 1} lines)")
|
||||
if in_git and not argv or base:
|
||||
rngs = changed_ranges(path, base)
|
||||
print("changed lines: " + (", ".join(rngs) if rngs else "whole file (new/untracked)"))
|
||||
if rows:
|
||||
print("outline (line range / start line):"); print("\n".join(" " + r for r in cap(rows, 60)))
|
||||
lint = linters(path, lang, text)
|
||||
if lint is None:
|
||||
print(f"compiler/linter: NOT CHECKED - no {'/'.join(TOOLS.get(lang, ['checker']))} installed for {lang}")
|
||||
else:
|
||||
print("compiler/linter: " + ("clean" if not lint else ""))
|
||||
if lint:
|
||||
print("\n".join(" " + l for l in cap(lint)))
|
||||
if hints:
|
||||
print("hints (verify before reporting):"); print("\n".join(" " + h for h in cap(hints)))
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main(sys.argv[1:])
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
---
|
||||
name: html-file-formats
|
||||
description: >
|
||||
Load, inspect and save files inside a single-file HTML app (browser, no
|
||||
server): images (png/jpg/webp), svg, stl, json, yaml, csv, xls/xlsx, docx,
|
||||
pdf and zip - which pinned CDN library to use, how to read the file the user
|
||||
picks, extract text/tables/pixels/geometry from it, and generate a download
|
||||
of your own data in that format. Use whenever a web app must open or export
|
||||
one of these formats; for .doc or other formats without a browser parser it
|
||||
says what to do instead.
|
||||
---
|
||||
|
||||
# File formats in a browser app
|
||||
|
||||
Everything runs in the page: the user picks a file (`<input type=file>`), you
|
||||
parse it in JS, and you save by generating bytes and triggering a download.
|
||||
Nothing is written to disk by the app itself - "save" always means download.
|
||||
Verified library versions and calls: [formats.md](./formats.md). Copy the
|
||||
import map / script tags from there; pin versions, never `@latest`.
|
||||
|
||||
| Format | Load (file -> data) | Save (data -> file) | Library | Notes |
|
||||
|---|---|---|---|---|
|
||||
| png / jpg / webp | `Image` + `<canvas>` -> pixels, size | `canvas.toBlob(cb, 'image/png')` | none | jpg has no alpha; `toBlob('image/jpeg', 0.9)` |
|
||||
| svg | `DOMParser` -> DOM (paths, sizes); rasterize via `Image` | serialize DOM / build string | none; three.js `SVGLoader` for extrusion | keep `viewBox`; CSS inside `<style>` |
|
||||
| stl | `STLLoader.parse(ArrayBuffer)` -> geometry | `STLExporter` | three.js addons | details: threejs-csg skill, loaders.md |
|
||||
| json | `JSON.parse(text)` | `JSON.stringify(data, null, 2)` | none | full recipe: save-load-json skill |
|
||||
| yaml | `yaml.load(text)` | `yaml.dump(obj)` | js-yaml 5.5.0 (ESM) | `load` is safe (no code execution) |
|
||||
| csv / tsv | `Papa.parse(text, {header, dynamicTyping})` | `Papa.unparse(rows)` | papaparse 5.7.0 | delimiter auto-detected; `encoding` for Latin-1 |
|
||||
| xlsx / xls | `XLSX.read(ArrayBuffer)` -> sheets -> json/csv | `XLSX.write(wb, {bookType})` | xlsx 0.18.5 (SheetJS) | reads legacy `.xls` too; formulas come as values + `f` |
|
||||
| docx | `mammoth.convertToHtml({arrayBuffer})` / `extractRawText` | `docx` 9.13.0 `Packer.toBlob` | mammoth 1.13.0, docx 9.13.0 | mammoth: text, headings, tables, images; no layout |
|
||||
| doc (legacy) | no browser parser | - | - | convert first: `soffice --headless --convert-to docx x.doc` (run_command), or ask for .docx |
|
||||
| pdf | pdf.js `getTextContent()` per page, render page to canvas | pdf-lib (precise, merge/modify) or jsPDF (quick text/images) | pdfjs-dist 6.4.299, pdf-lib 1.17.1, jspdf 4.2.1 | scanned PDFs have no text layer: render + OCR (tesseract.js) |
|
||||
| zip | `JSZip.loadAsync(ArrayBuffer)` | `zip.generateAsync({type:'blob'})` | jszip 3.10.2 | bundle a project: json + assets |
|
||||
|
||||
## Workflow
|
||||
1. Pick the row, copy its library line from formats.md into the page (import
|
||||
map entry or `<script>` tag) and the load/save snippet.
|
||||
2. Read the file as the right type: **text** for json/yaml/csv/svg,
|
||||
**ArrayBuffer** for everything binary (images, stl, xlsx, docx, pdf, zip).
|
||||
Helpers `pickFile`/`download` are in the single-file-webapp skill.
|
||||
3. Parse into plain data first (rows, records, geometry, text per page), THEN
|
||||
build app objects from it through the app's normal constructor. Never keep
|
||||
the library's document object as app state.
|
||||
4. Save = build from app state -> library -> `Blob` -> `download(name, blob)`.
|
||||
The button handler calls one `exportX()` function that RETURNS the Blob, so
|
||||
the browser-test skill can test it (`h.download('#btn')`).
|
||||
5. Test with real files: put small fixtures in `fixtures/` (a 2-row csv, a
|
||||
3-cell xlsx, a one-page pdf), drive the real buttons with `h.upload`,
|
||||
assert on the parsed data; then round-trip your own export through your
|
||||
own importer.
|
||||
|
||||
## Pitfalls
|
||||
- `FileReader.readAsText` guesses UTF-8; pass `'windows-1252'` for old CSV/TXT
|
||||
exports from Excel, or detect the BOM.
|
||||
- pdf.js **transfers** the ArrayBuffer you give it (it becomes empty). Pass
|
||||
`new Uint8Array(bytes)` or `bytes.slice()` if you need the bytes afterwards
|
||||
(e.g. to merge with pdf-lib).
|
||||
- pdf.js needs its worker: set `GlobalWorkerOptions.workerSrc` to the CDN
|
||||
worker URL of the SAME version, or every call fails with a worker error.
|
||||
- SheetJS dates: `cellDates: true` on read, else dates arrive as serial numbers.
|
||||
- Big files: parse in a `Worker` or at least `await` in chunks (`Papa.parse`
|
||||
has `step`/`chunk` callbacks); the UI freezes above a few MB otherwise.
|
||||
- `canvas.toBlob` on an image from another origin throws (tainted canvas):
|
||||
images must come from the user's file or the same origin.
|
||||
- Binary formats never go through `readAsText`; a docx/xlsx/pdf opened as text
|
||||
is garbage and the library will say "corrupt file".
|
||||
|
|
@ -0,0 +1,170 @@
|
|||
# Load / save snippets per format (browser)
|
||||
|
||||
All library calls below were run against the pinned versions. `pickFile(accept,
|
||||
asText, cb)` and `download(name, data, type)` are the helpers from the
|
||||
single-file-webapp skill (patterns.md); `download` accepts a string or a Blob.
|
||||
|
||||
## Libraries - copy what you need
|
||||
ESM (import map):
|
||||
```html
|
||||
<script type="importmap">{ "imports": {
|
||||
"js-yaml": "https://cdn.jsdelivr.net/npm/js-yaml@5.5.0/dist/js-yaml.mjs",
|
||||
"pdfjs": "https://cdn.jsdelivr.net/npm/pdfjs-dist@6.4.299/build/pdf.min.mjs",
|
||||
"docx": "https://cdn.jsdelivr.net/npm/docx@9.13.0/dist/index.mjs"
|
||||
}}</script>
|
||||
```
|
||||
Classic scripts (global objects `Papa`, `XLSX`, `mammoth`, `PDFLib`, `jspdf`, `JSZip`) - put
|
||||
them BEFORE your module script:
|
||||
```html
|
||||
<script src="https://cdn.jsdelivr.net/npm/papaparse@5.7.0/papaparse.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/xlsx@0.18.5/dist/xlsx.full.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/mammoth@1.13.0/mammoth.browser.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/pdf-lib@1.17.1/dist/pdf-lib.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/jspdf@4.2.1/dist/jspdf.umd.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/jszip@3.10.2/dist/jszip.min.js"></script>
|
||||
```
|
||||
|
||||
## Images (png / jpg / webp) - no library
|
||||
```js
|
||||
// load: file -> <img> -> pixels
|
||||
pickFile('image/*', false, (buf, name) => {
|
||||
const img = new Image();
|
||||
img.onload = () => {
|
||||
const c = Object.assign(document.createElement('canvas'), { width: img.width, height: img.height });
|
||||
const ctx = c.getContext('2d'); ctx.drawImage(img, 0, 0);
|
||||
const { data } = ctx.getImageData(0, 0, c.width, c.height); // RGBA bytes, row-major
|
||||
console.log(name, img.width, 'x', img.height, 'first pixel', [...data.slice(0, 4)]);
|
||||
URL.revokeObjectURL(img.src);
|
||||
};
|
||||
img.src = URL.createObjectURL(new Blob([buf]));
|
||||
});
|
||||
// save: canvas -> file
|
||||
canvas.toBlob(blob => download('out.png', blob), 'image/png');
|
||||
canvas.toBlob(blob => download('out.jpg', blob), 'image/jpeg', 0.9); // quality 0..1, no alpha
|
||||
```
|
||||
|
||||
## SVG - no library (three.js for extrusion)
|
||||
```js
|
||||
pickFile('.svg', true, text => {
|
||||
const svg = new DOMParser().parseFromString(text, 'image/svg+xml').documentElement;
|
||||
if (svg.querySelector('parsererror')) throw new Error('invalid SVG');
|
||||
const vb = svg.getAttribute('viewBox')?.split(/\s+|,/).map(Number); // [x, y, w, h]
|
||||
const paths = [...svg.querySelectorAll('path')].map(p => p.getAttribute('d'));
|
||||
// rasterize: draw into a canvas at any size
|
||||
const img = new Image(); img.onload = () => ctx.drawImage(img, 0, 0, 512, 512);
|
||||
img.src = 'data:image/svg+xml;charset=utf-8,' + encodeURIComponent(text);
|
||||
});
|
||||
// save: build the markup (keep viewBox so it scales) - or serialize a DOM you built
|
||||
const svgText = `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100"><circle cx="50" cy="50" r="40" fill="tomato"/></svg>`;
|
||||
download('shape.svg', svgText, 'image/svg+xml');
|
||||
// 3D: SVGLoader + ExtrudeGeometry -> threejs-csg skill, loaders.md
|
||||
```
|
||||
|
||||
## STL -> threejs-csg skill, loaders.md (`STLLoader.parse(arrayBuffer)`, `STLExporter`)
|
||||
|
||||
## JSON -> save-load-json skill
|
||||
|
||||
## YAML (js-yaml 5.5.0, ESM)
|
||||
```js
|
||||
import { load as yamlLoad, dump as yamlDump } from 'js-yaml';
|
||||
pickFile('.yaml,.yml', true, text => { const cfg = yamlLoad(text); /* plain objects/arrays */ });
|
||||
download('config.yaml', yamlDump(cfg, { noRefs: true, lineWidth: 100 }), 'text/yaml');
|
||||
```
|
||||
`load` never executes code (safe for untrusted files). Multi-document files: `loadAll(text)`.
|
||||
|
||||
## CSV / TSV (papaparse 5.7.0)
|
||||
```js
|
||||
pickFile('.csv,.tsv,.txt', true, text => {
|
||||
const { data, errors, meta } = Papa.parse(text, { header: true, dynamicTyping: true, skipEmptyLines: true });
|
||||
// data: [{col: value, ...}], meta.delimiter: ',' ';' or '\t' (auto), errors: [{row, message}]
|
||||
});
|
||||
download('table.csv', Papa.unparse(rows), 'text/csv'); // rows: array of objects or arrays
|
||||
// Excel-friendly: '' + Papa.unparse(rows, { delimiter: ';' }) (BOM + semicolon for German Excel)
|
||||
```
|
||||
Latin-1 files: `new TextDecoder('windows-1252').decode(arrayBuffer)` before parsing.
|
||||
|
||||
## XLSX / XLS (SheetJS xlsx 0.18.5)
|
||||
```js
|
||||
pickFile('.xlsx,.xls,.ods', false, buf => {
|
||||
const wb = XLSX.read(buf, { type: 'array', cellDates: true });
|
||||
for (const name of wb.SheetNames) {
|
||||
const ws = wb.Sheets[name];
|
||||
const rows = XLSX.utils.sheet_to_json(ws); // [{Header: value}], first row = headers
|
||||
const grid = XLSX.utils.sheet_to_json(ws, { header: 1 }); // [[cell, cell, ...]] raw grid
|
||||
const csv = XLSX.utils.sheet_to_csv(ws);
|
||||
const cell = ws['B2']; // {v: value, t: 'n'|'s'|'d'|'b', f: 'A2*2' (formula), w: formatted text}
|
||||
}
|
||||
});
|
||||
// save
|
||||
const wb = XLSX.utils.book_new();
|
||||
XLSX.utils.book_append_sheet(wb, XLSX.utils.json_to_sheet(rows), 'Data'); // or aoa_to_sheet([[...], [...]])
|
||||
const bytes = XLSX.write(wb, { type: 'array', bookType: 'xlsx' }); // 'xls' for legacy Excel, 'csv', 'ods'
|
||||
download('data.xlsx', new Blob([bytes], { type: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' }));
|
||||
```
|
||||
Formulas on write: `ws['C1'] = { t: 'n', f: 'A1+B1' }`. Column widths: `ws['!cols'] = [{ wch: 20 }]`.
|
||||
|
||||
## DOCX - read with mammoth 1.13.0, write with docx 9.13.0
|
||||
```js
|
||||
// read: text and simple HTML (headings, lists, tables, bold/italic, images as data URIs)
|
||||
pickFile('.docx', false, async buf => {
|
||||
const { value: html, messages } = await mammoth.convertToHtml({ arrayBuffer: buf });
|
||||
const { value: text } = await mammoth.extractRawText({ arrayBuffer: buf });
|
||||
// tables: parse the html with DOMParser and read <table><tr><td>
|
||||
});
|
||||
// write
|
||||
import { Document, Packer, Paragraph, TextRun, HeadingLevel, Table, TableRow, TableCell, ImageRun } from 'docx';
|
||||
const doc = new Document({ sections: [{ children: [
|
||||
new Paragraph({ text: 'Report', heading: HeadingLevel.HEADING_1 }),
|
||||
new Paragraph({ children: [new TextRun('Normal '), new TextRun({ text: 'bold', bold: true })] }),
|
||||
new Table({ rows: [new TableRow({ children: [new TableCell({ children: [new Paragraph('cell')] })] })] }),
|
||||
// new Paragraph({ children: [new ImageRun({ type: 'png', data: pngArrayBuffer, transformation: { width: 300, height: 200 } })] }),
|
||||
]}]});
|
||||
download('report.docx', await Packer.toBlob(doc));
|
||||
```
|
||||
`.doc` (Word 97-2003): no browser parser. Convert with LibreOffice via run_command
|
||||
(`soffice --headless --convert-to docx file.doc`) or ask for a .docx export.
|
||||
|
||||
## PDF - read with pdf.js 6.4.299, write with pdf-lib 1.17.1 or jsPDF 4.2.1
|
||||
```js
|
||||
import * as pdfjs from 'pdfjs';
|
||||
pdfjs.GlobalWorkerOptions.workerSrc = 'https://cdn.jsdelivr.net/npm/pdfjs-dist@6.4.299/build/pdf.worker.min.mjs';
|
||||
pickFile('.pdf', false, async buf => {
|
||||
const pdf = await pdfjs.getDocument({ data: new Uint8Array(buf) }).promise; // pass a COPY: pdf.js empties the buffer it gets
|
||||
for (let i = 1; i <= pdf.numPages; i++) {
|
||||
const page = await pdf.getPage(i);
|
||||
const text = (await page.getTextContent()).items.map(it => it.str).join(' '); // '' for scanned pages
|
||||
const vp = page.getViewport({ scale: 1.5 }); // render to a canvas
|
||||
const c = Object.assign(document.createElement('canvas'), { width: vp.width, height: vp.height });
|
||||
await page.render({ canvasContext: c.getContext('2d'), viewport: vp }).promise;
|
||||
}
|
||||
});
|
||||
// write (pdf-lib): exact placement, merge, fill forms, modify existing files
|
||||
const { PDFDocument, StandardFonts, rgb } = PDFLib;
|
||||
const out = await PDFDocument.create();
|
||||
const page = out.addPage([595, 842]); // A4 points
|
||||
const font = await out.embedFont(StandardFonts.Helvetica);
|
||||
page.drawText('Invoice 4711', { x: 50, y: 790, size: 14, font, color: rgb(0, 0, 0) });
|
||||
const png = await out.embedPng(pngBytes); page.drawImage(png, { x: 50, y: 500, width: 200, height: 150 });
|
||||
download('out.pdf', new Blob([await out.save()], { type: 'application/pdf' }));
|
||||
// merge
|
||||
const merged = await PDFDocument.create();
|
||||
for (const bytes of [a, b]) { const src = await PDFDocument.load(bytes); (await merged.copyPages(src, src.getPageIndices())).forEach(p => merged.addPage(p)); }
|
||||
// write (jsPDF): quick text/HTML-ish reports
|
||||
const doc = new jspdf.jsPDF(); doc.text('Hello', 20, 20); doc.addImage(canvas, 'PNG', 20, 30, 100, 60);
|
||||
download('report.pdf', doc.output('blob'));
|
||||
```
|
||||
Scanned PDFs: `getTextContent()` is empty - render the page and run OCR
|
||||
(tesseract.js 5: `Tesseract.recognize(canvas, 'eng')`), or say the file has no text layer.
|
||||
|
||||
## ZIP (jszip 3.10.2) - bundle a project with assets
|
||||
```js
|
||||
const zip = new JSZip();
|
||||
zip.file('scene.json', JSON.stringify(toData(), null, 2));
|
||||
zip.folder('textures').file('wood.png', pngBlob);
|
||||
download('project.zip', await zip.generateAsync({ type: 'blob' }));
|
||||
pickFile('.zip', false, async buf => {
|
||||
const z = await JSZip.loadAsync(buf);
|
||||
const scene = JSON.parse(await z.file('scene.json').async('string'));
|
||||
const tex = await z.file('textures/wood.png')?.async('blob');
|
||||
});
|
||||
```
|
||||
|
|
@ -0,0 +1,83 @@
|
|||
---
|
||||
name: save-load-json
|
||||
description: >
|
||||
Save and load an application's complete state as JSON (browser apps with
|
||||
download/upload, Python/Node programs with files): one state object, a
|
||||
serializer that saves EVERYTHING except an explicit runtime-only list, a
|
||||
loader that rebuilds through the normal constructors, a version field, and a
|
||||
round-trip check that fails when any field is lost. Use whenever a task says
|
||||
save/load, export/import, persist, project file, settings file, or when
|
||||
save/load "loses" data.
|
||||
---
|
||||
|
||||
# Save / load with JSON
|
||||
|
||||
The bug this skill prevents: a hand-written `toJSON` that copies a few fields
|
||||
(`id`, `type`) and silently drops the rest (`position`, `children`, ...). The
|
||||
user saves, loads, and the scene is wrong. The fix is a rule, not care:
|
||||
|
||||
> **Save everything by default.** Serialize the whole state object and
|
||||
> subtract an explicit `RUNTIME` list (meshes, DOM nodes, caches, selection).
|
||||
> A forgotten field is then saved, not lost.
|
||||
|
||||
## Design (do this before writing the app)
|
||||
1. **One state object** holds all data the user would expect to get back:
|
||||
objects, their parameters and transforms, nesting, settings (grid size,
|
||||
units, colors), view (camera) if restoring it is useful.
|
||||
2. **Data vs runtime.** Each object is a plain record (`{id, type, params,
|
||||
position, rotation, scale, color, hole, children}`); runtime things
|
||||
(`mesh`, `element`, `texture`, listeners) live in a side map
|
||||
`runtime.get(id)` or in fields named in `RUNTIME`. Never store a live object
|
||||
inside the data record if you can avoid it.
|
||||
3. **One constructor path.** `fromRecord(rec)` creates a runtime object from a
|
||||
data record. The "add shape" button, paste, duplicate AND load all call
|
||||
`fromRecord` - so load can't behave differently from add.
|
||||
4. **Format:** `{ "version": 1, "app": "<name>", "state": {...} }`. Bump
|
||||
`version` when the shape changes; `load` migrates older versions
|
||||
(`migrate(data)`) instead of failing. Numbers as numbers (not strings),
|
||||
arrays for vectors (`[x, y, z]`), ISO strings for dates, no `NaN`/`Infinity`
|
||||
(JSON has none - clamp or null them).
|
||||
5. **Round trip is the test:** `deepEqual(toData(fromData(toData(state))),
|
||||
toData(state))`, and the field-completeness check below. Run both before
|
||||
wiring any button.
|
||||
|
||||
## Browser app
|
||||
- `toData()` returns plain data; `download('x.json', JSON.stringify(toData(), null, 2))`
|
||||
is the ENTIRE save handler. `pickFile('.json', true, text => load(JSON.parse(text)))`
|
||||
is the entire load handler. No second serializer in a handler, ever.
|
||||
- Test the real buttons with the browser-test skill (`h.download('#save')`,
|
||||
`h.upload('#load', 'fixtures/scene.json')`) and compare the file's fields to
|
||||
the live state - that catches the handler calling the wrong function.
|
||||
- Snippets: [js.md](./js.md) (`toData`/`load` with RUNTIME subtraction,
|
||||
migrate, Set/Map/typed arrays, the completeness check, fixture generation).
|
||||
|
||||
## Python / Node programs
|
||||
- Python: dataclasses + `dataclasses.asdict` for the data side, a `RUNTIME`
|
||||
tuple of field names to drop, `json.dump(..., indent=2, ensure_ascii=False)`,
|
||||
write to a temp file and `os.replace` so a crash never leaves a half file.
|
||||
Snippets and the round-trip test: [python.md](./python.md).
|
||||
- Node: same rules; `JSON.stringify(data, null, 2)` + `fs.writeFileSync(tmp)` +
|
||||
`fs.renameSync`.
|
||||
|
||||
## Completeness check (run it, don't eyeball it)
|
||||
`check_roundtrip.py` compares a saved JSON file against a list of the fields
|
||||
the live state has (dumped from the app via the test hook or a one-liner) and
|
||||
prints every field present in the state but missing from the file:
|
||||
```
|
||||
python <skill-dir>/check_roundtrip.py saved.json state_fields.json
|
||||
```
|
||||
Details and how to produce `state_fields.json` in [js.md](./js.md#completeness)
|
||||
and [python.md](./python.md#completeness).
|
||||
|
||||
## Pitfalls
|
||||
- `JSON.stringify(state)` with live objects: cyclic structure error, or megabytes
|
||||
of mesh data - that's the signal to split data from runtime (rule 2).
|
||||
- Loading replaces the whole state; clear selection/undo history or they point
|
||||
at ids that no longer exist.
|
||||
- Nested groups/children: recursion in both directions, and ids must stay
|
||||
unique after load (`nextId = max(ids) + 1`).
|
||||
- Floats: don't round on save; round only for display.
|
||||
- Binary data (images, STL buffers): base64 in the JSON only if small; otherwise
|
||||
a zip or separate files, with file names in the JSON.
|
||||
- Don't let a load failure wipe the current state: parse and validate first,
|
||||
then swap.
|
||||
|
|
@ -0,0 +1,76 @@
|
|||
#!/usr/bin/env python3
|
||||
"""check_roundtrip.py - is every field of the live state in the saved file?
|
||||
|
||||
python check_roundtrip.py saved.json state_fields.json [--runtime selection,_cache]
|
||||
|
||||
saved.json the file the app wrote (any JSON; a top-level "state" key is unwrapped)
|
||||
state_fields.json JSON array of field paths of the LIVE state, e.g.
|
||||
["objects[].id", "objects[].position[]", "settings.grid", "selection"]
|
||||
(how to produce it: js.md / python.md, section "Completeness")
|
||||
--runtime comma-separated paths that are intentionally not saved
|
||||
|
||||
Exit 1 if a state field is missing from the file (and not in --runtime), with the list.
|
||||
Also lists file fields that the state does not have (stale keys from old versions).
|
||||
"""
|
||||
import json, sys
|
||||
|
||||
|
||||
def paths(obj, prefix=""):
|
||||
if isinstance(obj, dict):
|
||||
if not obj:
|
||||
yield prefix
|
||||
for k, v in obj.items():
|
||||
yield from paths(v, f"{prefix}.{k}" if prefix else k)
|
||||
elif isinstance(obj, list):
|
||||
if not obj:
|
||||
yield prefix # empty array: can't know the element shape
|
||||
for v in obj:
|
||||
yield from paths(v, prefix + "[]")
|
||||
else:
|
||||
yield prefix
|
||||
|
||||
|
||||
def normalize(p):
|
||||
"""'objects[].children[].id' and 'objects[].id' describe the same record type when
|
||||
children hold the same records; compare on the path with repeated '[]' segments kept."""
|
||||
return p
|
||||
|
||||
|
||||
def main(argv):
|
||||
runtime = set()
|
||||
if "--runtime" in argv:
|
||||
i = argv.index("--runtime"); runtime = set(argv[i + 1].split(",")); del argv[i:i + 2]
|
||||
if len(argv) != 2:
|
||||
print(__doc__); return 2
|
||||
saved = json.load(open(argv[0], encoding="utf-8"))
|
||||
if isinstance(saved, dict) and "state" in saved and isinstance(saved["state"], dict):
|
||||
saved = saved["state"]
|
||||
want = set(json.load(open(argv[1], encoding="utf-8")))
|
||||
have = set(paths(saved))
|
||||
# a field counts as present if the file has it at that path or any ancestor is an empty container
|
||||
def present(p):
|
||||
if p in have:
|
||||
return True
|
||||
parts = p.replace("[]", ".[]").split(".")
|
||||
for n in range(len(parts) - 1, 0, -1):
|
||||
anc = ".".join(parts[:n]).replace(".[]", "[]")
|
||||
if anc in have and any(h == anc for h in have): # ancestor exists as a leaf = empty list/dict
|
||||
return True
|
||||
return False
|
||||
missing = sorted(p for p in want if not present(p) and p not in runtime
|
||||
and not any(p == r or p.startswith(r + ".") or p.startswith(r + "[]") for r in runtime))
|
||||
extra = sorted(h for h in have if h not in want)
|
||||
if missing:
|
||||
print(f"MISSING in the file ({len(missing)}): fields the live state has but the save does not")
|
||||
for p in missing:
|
||||
print(" -", p)
|
||||
if extra:
|
||||
print(f"only in the file ({len(extra)}): not in the live state (old keys? wrong source?)")
|
||||
for p in extra:
|
||||
print(" -", p)
|
||||
print("ok - every state field is in the file" if not missing else f"{len(missing)} field(s) lost on save")
|
||||
return 1 if missing else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
|
|
@ -0,0 +1,116 @@
|
|||
# Save / load in a browser app (ES module)
|
||||
|
||||
## State layout
|
||||
```js
|
||||
const state = {
|
||||
version: 1,
|
||||
settings: { grid: 1, units: 'mm' },
|
||||
objects: [], // data records only: {id, type, params, position, rotation, scale, color, hole, children}
|
||||
nextId: 1,
|
||||
selection: new Set(), // runtime: not saved
|
||||
};
|
||||
const runtime = new Map(); // id -> { mesh, ... } (never inside the records)
|
||||
const RUNTIME = ['selection']; // top-level fields that are NOT saved
|
||||
```
|
||||
|
||||
## toData / load - save everything except RUNTIME
|
||||
```js
|
||||
function toData() {
|
||||
const data = { ...state };
|
||||
for (const k of RUNTIME) delete data[k];
|
||||
assertPlain(data); // on the LIVE data: meshes, cycles, class instances -> error now, not a broken file
|
||||
return { version: state.version, app: 'myapp', state: structuredClone(data) };
|
||||
}
|
||||
function assertPlain(v, path = 'state', seen = new Set()) {
|
||||
if (v === null || typeof v !== 'object') { if (typeof v === 'number' && !Number.isFinite(v)) throw new Error(`${path} is ${v} - JSON has no NaN/Infinity`); return; }
|
||||
if (seen.has(v)) throw new Error(`${path} is cyclic - a runtime object leaked into the data (add it to RUNTIME or the runtime map)`);
|
||||
seen.add(v);
|
||||
const proto = Object.getPrototypeOf(v);
|
||||
if (proto !== Object.prototype && proto !== Array.prototype && proto !== null)
|
||||
throw new Error(`${path} is a ${v.constructor?.name ?? 'class instance'} - store plain data, keep the object in the runtime map`);
|
||||
for (const k of Object.keys(v)) assertPlain(v[k], `${path}.${k}`, seen);
|
||||
}
|
||||
|
||||
function fromRecord(rec) { // the ONE constructor path (add, paste, load)
|
||||
const obj = { children: [], ...rec }; // defaults first, record wins
|
||||
runtime.set(obj.id, { mesh: buildMesh(obj) });
|
||||
obj.children.forEach(fromRecord);
|
||||
return obj;
|
||||
}
|
||||
|
||||
function load(data) {
|
||||
data = migrate(data); // old versions -> current
|
||||
validate(data); // throws -> current state untouched
|
||||
clearRuntime();
|
||||
Object.assign(state, data.state); // replaces objects, settings, nextId, ...
|
||||
state.selection = new Set(); // re-create runtime fields
|
||||
state.objects = state.objects.map(fromRecord);
|
||||
state.nextId = Math.max(state.nextId ?? 1, ...allIds(state.objects).map(i => i + 1)); // never reuse an id
|
||||
render();
|
||||
}
|
||||
const allIds = objs => objs.flatMap(o => [o.id, ...allIds(o.children ?? [])]);
|
||||
|
||||
function migrate(data) {
|
||||
if (data.version === undefined) data = { version: 0, state: data }; // very old: bare state
|
||||
if (data.version < 1) { data.state.objects.forEach(o => { o.scale ??= [1, 1, 1]; }); data.version = 1; }
|
||||
return data;
|
||||
}
|
||||
function validate(data) {
|
||||
if (!data?.state || !Array.isArray(data.state.objects)) throw new Error('not a myapp file');
|
||||
}
|
||||
```
|
||||
`assertPlain` is what turns "the JSON is silently wrong" into an error at save
|
||||
time: a three.js mesh, a cyclic parent link or a `Set` in the data record fails
|
||||
loudly with the path. `Map`/`Set`/typed arrays survive `structuredClone` but
|
||||
`JSON.stringify` turns them into `{}` - convert those explicitly:
|
||||
```js
|
||||
// in toData(): data.tags = [...state.tags]; // Set -> array
|
||||
// in load(): state.tags = new Set(data.state.tags);
|
||||
// Float32Array -> Array.from(a); Date -> d.toISOString(); back: new Date(s)
|
||||
```
|
||||
|
||||
## Buttons - nothing but a call
|
||||
```js
|
||||
$('save').addEventListener('click', () => download('scene.json', JSON.stringify(toData(), null, 2), 'application/json'));
|
||||
$('load').addEventListener('click', () => pickFile('.json', true, text => {
|
||||
try { load(JSON.parse(text)); } catch (e) { alert('Load failed: ' + e.message); }
|
||||
}));
|
||||
```
|
||||
(`download`/`pickFile` are in the single-file-webapp skill, patterns.md.)
|
||||
|
||||
## Round-trip test (browser-test skill, through the REAL buttons)
|
||||
```js
|
||||
module.exports = async (h) => {
|
||||
await h.eval(() => { __app.add('box', { size: 2 }); __app.add('cylinder', { r: 0.5, h: 3 }); });
|
||||
await h.eval(() => { const o = __app.state.objects[1]; o.position = [3, 0, 1]; o.hole = true; });
|
||||
const before = await h.eval(() => JSON.stringify(__app.toData()));
|
||||
const f = await h.download('#save'); // real export path
|
||||
if (f.text !== JSON.stringify(JSON.parse(before), null, 2)) throw new Error('save button wrote something else than toData()');
|
||||
require('fs').writeFileSync('fixtures/scene.json', f.text);
|
||||
await h.eval(() => __app.reset());
|
||||
await h.upload('#load', 'fixtures/scene.json'); // real import path
|
||||
const after = await h.eval(() => JSON.stringify(__app.toData()));
|
||||
if (after !== before) throw new Error('round trip differs:\n' + before + '\n' + after);
|
||||
};
|
||||
```
|
||||
Expose `toData`, `add`, `reset` on `window.__app`.
|
||||
|
||||
## Completeness
|
||||
Dump the field paths of the live state (runtime fields included) once:
|
||||
```js
|
||||
const fields = await h.eval(() => {
|
||||
const out = new Set();
|
||||
(function walk(v, p) {
|
||||
if (Array.isArray(v)) { v.forEach(x => walk(x, p + '[]')); return; }
|
||||
if (v && typeof v === 'object' && !(v instanceof Set) && !(v instanceof Map)) {
|
||||
for (const k of Object.keys(v)) walk(v[k], p ? p + '.' + k : k); return;
|
||||
}
|
||||
out.add(p);
|
||||
})(__app.state, '');
|
||||
return [...out];
|
||||
});
|
||||
require('fs').writeFileSync('state_fields.json', JSON.stringify(fields, null, 2));
|
||||
```
|
||||
Then `python <skill-dir>/check_roundtrip.py fixtures/scene.json state_fields.json`
|
||||
lists every state field that the file lacks. Each one is either a RUNTIME
|
||||
field (add it to the list in the app, so the omission is explicit) or a bug.
|
||||
|
|
@ -0,0 +1,109 @@
|
|||
# Save / load in Python
|
||||
|
||||
## Data classes, one `RUNTIME` list
|
||||
```python
|
||||
from dataclasses import dataclass, field, asdict, fields
|
||||
import json, os, tempfile
|
||||
from pathlib import Path
|
||||
|
||||
VERSION = 1
|
||||
|
||||
@dataclass
|
||||
class Item:
|
||||
id: int
|
||||
type: str
|
||||
params: dict = field(default_factory=dict)
|
||||
position: list = field(default_factory=lambda: [0.0, 0.0, 0.0])
|
||||
children: list = field(default_factory=list) # nested Items
|
||||
|
||||
@dataclass
|
||||
class State:
|
||||
items: list = field(default_factory=list)
|
||||
settings: dict = field(default_factory=lambda: {"grid": 1.0})
|
||||
next_id: int = 1
|
||||
selection: set = field(default_factory=set) # runtime only
|
||||
_cache: dict = field(default_factory=dict) # runtime only
|
||||
|
||||
RUNTIME = ("selection", "_cache") # the ONLY fields not saved
|
||||
```
|
||||
|
||||
## to_data / load
|
||||
```python
|
||||
def to_data(state: State) -> dict:
|
||||
data = asdict(state) # everything, recursively
|
||||
for k in RUNTIME:
|
||||
data.pop(k, None)
|
||||
return {"version": VERSION, "app": "myapp", "state": data}
|
||||
|
||||
def from_record(rec: dict) -> Item: # the ONE constructor path
|
||||
rec = {**rec, "children": [from_record(c) for c in rec.get("children", [])]}
|
||||
return Item(**rec)
|
||||
|
||||
def load(data: dict) -> State:
|
||||
data = migrate(data)
|
||||
st = data["state"]
|
||||
state = State(items=[from_record(r) for r in st["items"]],
|
||||
settings=st.get("settings", {}), next_id=st.get("next_id", 1))
|
||||
state.next_id = max(state.next_id, max(_ids(state.items), default=0) + 1) # never reuse an id
|
||||
return state
|
||||
|
||||
def _ids(items):
|
||||
for it in items:
|
||||
yield it.id
|
||||
yield from _ids(it.children)
|
||||
|
||||
def migrate(data: dict) -> dict:
|
||||
if "version" not in data:
|
||||
data = {"version": 0, "state": data}
|
||||
if data["version"] < 1:
|
||||
for r in data["state"]["items"]:
|
||||
r.setdefault("position", [0.0, 0.0, 0.0])
|
||||
data["version"] = 1
|
||||
return data
|
||||
```
|
||||
|
||||
## Files: atomic write, explicit encoding
|
||||
```python
|
||||
def save_file(state: State, path: Path) -> None:
|
||||
tmp = tempfile.NamedTemporaryFile("w", dir=path.parent, suffix=".tmp", delete=False, encoding="utf-8")
|
||||
with tmp:
|
||||
json.dump(to_data(state), tmp, indent=2, ensure_ascii=False, allow_nan=False)
|
||||
os.replace(tmp.name, path) # never a half-written file
|
||||
|
||||
def load_file(path: Path) -> State:
|
||||
return load(json.loads(path.read_text(encoding="utf-8")))
|
||||
```
|
||||
`allow_nan=False` makes `NaN`/`inf` an error at save time instead of a file no
|
||||
other parser can read. For `Path`, `datetime`, `set`, `Decimal` use
|
||||
`json.dump(..., default=_enc)` with `_enc` returning `str(o)`, `o.isoformat()`,
|
||||
`sorted(o)`, `float(o)`, and convert back in `from_record`.
|
||||
|
||||
## Round-trip test (pytest)
|
||||
```python
|
||||
def test_roundtrip(tmp_path):
|
||||
s = State(items=[Item(1, "box", {"size": 2}, [1, 2, 3], [Item(2, "hole", {"r": 0.5})])], next_id=3)
|
||||
s.selection.add(1)
|
||||
save_file(s, tmp_path / "a.json")
|
||||
s2 = load_file(tmp_path / "a.json")
|
||||
assert to_data(s2) == to_data(s) # data equal
|
||||
assert s2.selection == set() # runtime rebuilt, not loaded
|
||||
assert s2.next_id == 3
|
||||
```
|
||||
|
||||
## Completeness
|
||||
```python
|
||||
def field_paths(obj, prefix=""):
|
||||
"""All leaf paths of a live state, dataclasses included."""
|
||||
if hasattr(obj, "__dataclass_fields__"):
|
||||
obj = {f.name: getattr(obj, f.name) for f in fields(obj)}
|
||||
if isinstance(obj, dict):
|
||||
for k, v in obj.items():
|
||||
yield from field_paths(v, f"{prefix}.{k}" if prefix else k)
|
||||
elif isinstance(obj, (list, tuple, set)):
|
||||
for v in obj:
|
||||
yield from field_paths(v, prefix + "[]")
|
||||
else:
|
||||
yield prefix
|
||||
json.dump(sorted(set(field_paths(state))), open("state_fields.json", "w"), indent=2)
|
||||
```
|
||||
then `python <skill-dir>/check_roundtrip.py a.json state_fields.json`.
|
||||
|
|
@ -0,0 +1,91 @@
|
|||
---
|
||||
name: scientific-deep-research
|
||||
description: >
|
||||
Literature research on a scientific question: find relevant papers (arXiv,
|
||||
OpenAlex, Semantic Scholar), prefer the peer-reviewed version of arXiv
|
||||
preprints (Crossref, dblp), download the PDFs, extract and read the text,
|
||||
and write research.md with structured per-paper summaries, a synthesis and
|
||||
a references.bib in BibTeX. Use for "survey the literature", "find papers
|
||||
on", "state of the art", "related work", or any task that needs citations.
|
||||
---
|
||||
|
||||
# Scientific deep research
|
||||
|
||||
Output: `research.md` (summaries + synthesis), `references.bib`, and the PDFs
|
||||
in `papers/`. Every claim in research.md is tied to a paper you actually
|
||||
opened; every BibTeX entry comes from a resolver, never from memory.
|
||||
|
||||
Tool: [papers.py](./papers.py) (stdlib only; `pip install pymupdf` for
|
||||
`text`). Copy it into the project: `cp <skill-dir>/papers.py .`. Be polite to
|
||||
the APIs: one call per second, `CONTACT_EMAIL=you@uni.de` in the environment.
|
||||
|
||||
## Procedure
|
||||
1. **Frame the question** in one sentence plus 3-6 search phrases (synonyms,
|
||||
method names, the application). Write them at the top of research.md.
|
||||
2. **Search:** `python papers.py search "<phrase>" --max 25 [--since 2018]`
|
||||
for each phrase. Merged, deduplicated candidates land in `candidates.json`,
|
||||
ranked by citations (recent papers have few - don't drop them for that).
|
||||
3. **Screen** by title + abstract (`candidates.json`): keep 8-15 papers that
|
||||
answer the question, including 2-3 recent ones and the most-cited classic.
|
||||
Note why each was kept in one line. Follow references of the best paper if
|
||||
coverage is thin (`search` with its key terms).
|
||||
4. **Resolve the citable version:** `python papers.py resolve <arxiv id | DOI | "title">`
|
||||
for every kept paper. For an arXiv preprint it looks for the journal or
|
||||
conference version (Semantic Scholar -> Crossref -> dblp) and prints which
|
||||
to cite. Cite the published version when one exists; otherwise cite the
|
||||
preprint and mark it `(preprint)` in research.md.
|
||||
5. **BibTeX:** `python papers.py bibtex <DOI | arXiv id | dblp key>` -> append
|
||||
to `references.bib`. Keys are `<lastname><year><firstword>`; use exactly
|
||||
these keys in the text as `[@key]`. One entry per paper, no duplicates.
|
||||
6. **Download and read:** `python papers.py download <arxiv id | DOI> --out papers/`
|
||||
(arXiv PDF, or the open-access copy OpenAlex knows; otherwise say "no OA
|
||||
PDF" and work from the abstract, marked as such). Then
|
||||
`python papers.py text papers/x.pdf` and read the .txt with `read_file`
|
||||
in ranges: abstract, introduction, method, results tables, conclusion.
|
||||
Don't read reference lists.
|
||||
7. **Summarize each paper** with the template below - numbers from the
|
||||
results, not adjectives. 120-200 words each.
|
||||
8. **Synthesize:** what the field agrees on, where results conflict, which
|
||||
method fits the question, open problems. Reference papers by key.
|
||||
9. **Verify before finishing:** every `[@key]` in research.md exists in
|
||||
references.bib and vice versa (`grep -o '@\w*' research.md | sort -u`
|
||||
against `grep -o '^@\w*{\K[^,]*' references.bib`); every summary names
|
||||
the version read (published / preprint / abstract only).
|
||||
|
||||
## research.md template
|
||||
```markdown
|
||||
# <Question>
|
||||
Date, search phrases, databases, number screened / kept.
|
||||
|
||||
## Summary of findings (10-15 lines)
|
||||
...with [@key] references...
|
||||
|
||||
## Papers
|
||||
### <Title> [@key]
|
||||
Authors, venue + year (or "arXiv preprint"), DOI/arXiv, read: full text | abstract only
|
||||
- **Problem:** ...
|
||||
- **Method:** ...
|
||||
- **Data / setup:** ...
|
||||
- **Results:** the numbers (metric, value, baseline, dataset)
|
||||
- **Limitations:** stated by the authors or evident
|
||||
- **Relevance to the question:** ...
|
||||
|
||||
## Synthesis
|
||||
Agreement, contradictions, best-fit method, open questions.
|
||||
|
||||
## Not covered
|
||||
Papers without accessible PDF, areas not searched.
|
||||
|
||||
## References
|
||||
See references.bib (keys above).
|
||||
```
|
||||
|
||||
## Rules
|
||||
- No citation without a resolved record. If a resolver finds nothing, the
|
||||
paper is "unverified" and goes into "Not covered", not into the text.
|
||||
- Never "improve" numbers from memory; quote what the PDF says and the page.
|
||||
- Preprint vs. published: results can differ between versions - say which
|
||||
one you read.
|
||||
- Keep the PDFs and `candidates.json`; the user may want to continue.
|
||||
- 10 well-read papers beat 40 abstracts. Stop searching when new queries
|
||||
return papers you already have.
|
||||
|
|
@ -0,0 +1,366 @@
|
|||
#!/usr/bin/env python3
|
||||
"""papers.py - find, resolve, download and cite scientific papers (stdlib only).
|
||||
|
||||
python papers.py search "boolean operations on triangle meshes" --max 25 [--since 2018]
|
||||
python papers.py resolve 1706.03762 # arXiv id, DOI or "title": find the peer-reviewed version
|
||||
python papers.py bibtex 10.1145/3306346.3322959 # DOI / arXiv id / dblp key -> BibTeX entry
|
||||
python papers.py download 1706.03762 --out papers/ # PDF (arXiv or open-access copy)
|
||||
python papers.py text papers/vaswani2017attention.pdf [--pages 12] # PDF -> .txt (pymupdf)
|
||||
|
||||
search merges arXiv, OpenAlex and Semantic Scholar; prints a ranked table and writes candidates.json.
|
||||
resolve prints, for one paper, the best version: journal/conference (DOI, venue, year) if one
|
||||
exists, otherwise the arXiv preprint - plus the BibTeX key you should use.
|
||||
bibtex prints a cleaned BibTeX entry with a stable key <lastname><year><firstword>.
|
||||
All network calls have a 20 s timeout and identify themselves; be polite (<= 1 request/s).
|
||||
Sources: export.arxiv.org (Atom), api.openalex.org, api.semanticscholar.org, api.crossref.org,
|
||||
dblp.org, doi.org (content negotiation). Set CONTACT_EMAIL for the Crossref/OpenAlex polite pools.
|
||||
"""
|
||||
import json, os, re, sys, time, unicodedata, urllib.parse, urllib.request
|
||||
import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
|
||||
UA = f"papers.py/1.0 (mailto:{os.environ.get('CONTACT_EMAIL', 'research@example.org')})"
|
||||
ARXIV_RE = re.compile(r"(?:arxiv[:/]|abs/|pdf/)?(\d{4}\.\d{4,5})(?:v\d+)?", re.I)
|
||||
DOI_RE = re.compile(r"\b(10\.\d{4,9}/[^\s\"<>]+)", re.I)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- helpers
|
||||
def get(url, accept="application/json", retries=2):
|
||||
req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept": accept})
|
||||
for attempt in range(retries + 1):
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=20) as r:
|
||||
return r.read().decode("utf-8", "replace")
|
||||
except Exception as e: # noqa: BLE001
|
||||
if attempt == retries:
|
||||
print(f" ! {url[:80]}...: {e}", file=sys.stderr)
|
||||
return None
|
||||
time.sleep(2 * (attempt + 1))
|
||||
|
||||
|
||||
def norm_title(t):
|
||||
t = unicodedata.normalize("NFKD", t or "").encode("ascii", "ignore").decode()
|
||||
return re.sub(r"[^a-z0-9]+", " ", t.lower()).strip()
|
||||
|
||||
|
||||
def bib_key(authors, year, title):
|
||||
last = (authors[0].split()[-1] if authors else "anon")
|
||||
last = re.sub(r"[^a-z]", "", unicodedata.normalize("NFKD", last).encode("ascii", "ignore").decode().lower()) or "anon"
|
||||
stop = {"a", "an", "the", "on", "of", "for", "and", "to", "in", "with", "towards", "toward"}
|
||||
first = next((w for w in re.findall(r"[a-z0-9]+", (title or "").lower()) if w not in stop), "paper")
|
||||
return f"{last}{year or 'nd'}{first}"
|
||||
|
||||
|
||||
def is_arxiv_doi(doi):
|
||||
return bool(doi) and doi.lower().startswith("10.48550/arxiv")
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- parsers (pure: testable offline)
|
||||
def parse_arxiv(atom):
|
||||
ns = {"a": "http://www.w3.org/2005/Atom", "x": "http://arxiv.org/schemas/atom"}
|
||||
out = []
|
||||
for e in ET.fromstring(atom).findall("a:entry", ns):
|
||||
aid = ARXIV_RE.search(e.findtext("a:id", "", ns))
|
||||
out.append({
|
||||
"arxiv": aid.group(1) if aid else None,
|
||||
"title": " ".join(e.findtext("a:title", "", ns).split()),
|
||||
"authors": [a.findtext("a:name", "", ns) for a in e.findall("a:author", ns)],
|
||||
"year": (e.findtext("a:published", "", ns) or "")[:4],
|
||||
"abstract": " ".join(e.findtext("a:summary", "", ns).split()),
|
||||
"doi": (e.findtext("x:doi", None, ns) or None),
|
||||
"venue": (e.findtext("x:journal_ref", None, ns) or None), # authors' own note, e.g. "NeurIPS 2017"
|
||||
"pdf": f"https://arxiv.org/pdf/{aid.group(1)}" if aid else None,
|
||||
"source": "arxiv",
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def parse_openalex(js):
|
||||
out = []
|
||||
for w in json.loads(js).get("results", []):
|
||||
ids = w.get("ids", {}) or {}
|
||||
doi = (w.get("doi") or "").replace("https://doi.org/", "") or None
|
||||
loc = w.get("primary_location") or {}
|
||||
src = loc.get("source") or {}
|
||||
best = w.get("best_oa_location") or {}
|
||||
# abstract comes as an inverted index
|
||||
inv = w.get("abstract_inverted_index") or {}
|
||||
words = sorted((p, t) for t, ps in inv.items() for p in ps)
|
||||
out.append({
|
||||
"arxiv": next((m.group(1) for u in (best.get("pdf_url"), best.get("landing_page_url"))
|
||||
if u and "arxiv.org" in u and (m := ARXIV_RE.search(u))), None),
|
||||
"title": w.get("title") or "",
|
||||
"authors": [a.get("author", {}).get("display_name", "") for a in w.get("authorships", [])],
|
||||
"year": str(w.get("publication_year") or ""),
|
||||
"abstract": " ".join(t for _, t in words),
|
||||
"doi": doi,
|
||||
"venue": src.get("display_name"),
|
||||
"type": w.get("type"), # article, preprint, book-chapter, ...
|
||||
"citations": w.get("cited_by_count", 0),
|
||||
"pdf": best.get("pdf_url"),
|
||||
"source": "openalex",
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def parse_s2(js):
|
||||
data = json.loads(js)
|
||||
items = data.get("data", [data]) if isinstance(data, dict) else data
|
||||
out = []
|
||||
for p in items:
|
||||
if not p:
|
||||
continue
|
||||
ext = p.get("externalIds") or {}
|
||||
pv = p.get("publicationVenue") or {}
|
||||
out.append({
|
||||
"arxiv": ext.get("ArXiv"),
|
||||
"title": p.get("title") or "",
|
||||
"authors": [a.get("name", "") for a in p.get("authors", [])],
|
||||
"year": str(p.get("year") or ""),
|
||||
"abstract": p.get("abstract") or "",
|
||||
"doi": ext.get("DOI"),
|
||||
"venue": p.get("venue") or pv.get("name"),
|
||||
"venue_type": pv.get("type"), # journal / conference
|
||||
"types": p.get("publicationTypes") or [], # JournalArticle, Conference, ...
|
||||
"citations": p.get("citationCount", 0),
|
||||
"pdf": (p.get("openAccessPdf") or {}).get("url"),
|
||||
"source": "s2",
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def parse_crossref(js, want_title):
|
||||
"""Best Crossref item whose title matches (journal/proceedings only), else None."""
|
||||
want = norm_title(want_title)
|
||||
for it in json.loads(js).get("message", {}).get("items", []):
|
||||
title = (it.get("title") or [""])[0]
|
||||
if norm_title(title) != want or it.get("type") not in ("journal-article", "proceedings-article", "book-chapter"):
|
||||
continue
|
||||
date = (it.get("published-print") or it.get("published-online") or it.get("issued") or {}).get("date-parts", [[None]])[0][0]
|
||||
return {
|
||||
"doi": it.get("DOI"), "title": title, "year": str(date or ""),
|
||||
"venue": (it.get("container-title") or [None])[0] or (it.get("event") or {}).get("name"),
|
||||
"type": it.get("type"),
|
||||
"authors": [f"{a.get('given', '')} {a.get('family', '')}".strip() for a in it.get("author", [])],
|
||||
"source": "crossref",
|
||||
}
|
||||
return None
|
||||
|
||||
|
||||
def parse_dblp(js, want_title):
|
||||
"""dblp hit for the title, preferring the conference/journal record over the CoRR (arXiv) one."""
|
||||
want = norm_title(want_title)
|
||||
hits = [h.get("info", {}) for h in json.loads(js).get("result", {}).get("hits", {}).get("hit", [])]
|
||||
hits = [h for h in hits if norm_title(h.get("title", "").rstrip(".")) == want]
|
||||
hits.sort(key=lambda h: (h.get("venue") == "CoRR", h.get("type") not in ("Conference and Workshop Papers", "Journal Articles")))
|
||||
if not hits:
|
||||
return None
|
||||
h = hits[0]
|
||||
return {"key": h.get("key"), "venue": h.get("venue"), "year": h.get("year"), "type": h.get("type"),
|
||||
"doi": h.get("doi"), "bib": f"https://dblp.org/rec/{h.get('key')}.bib", "source": "dblp"}
|
||||
|
||||
|
||||
def merge(cands):
|
||||
"""Deduplicate by DOI / arXiv id / normalized title; keep the most informative record."""
|
||||
seen, out = {}, []
|
||||
for c in cands:
|
||||
keys = [k for k in (c.get("doi") and c["doi"].lower(), c.get("arxiv"), norm_title(c["title"])) if k]
|
||||
hit = next((seen[k] for k in keys if k in seen), None)
|
||||
if hit is None:
|
||||
hit = dict(c); out.append(hit)
|
||||
else:
|
||||
for k, v in c.items():
|
||||
if v and not hit.get(k):
|
||||
hit[k] = v
|
||||
hit["citations"] = max(hit.get("citations") or 0, c.get("citations") or 0)
|
||||
for k in keys:
|
||||
seen[k] = hit
|
||||
return out
|
||||
|
||||
|
||||
def arxiv_bibtex(rec):
|
||||
key = bib_key(rec["authors"], rec["year"], rec["title"])
|
||||
return (f"@misc{{{key},\n title = {{{rec['title']}}},\n author = {{{' and '.join(rec['authors'])}}},\n"
|
||||
f" year = {{{rec['year']}}},\n eprint = {{{rec['arxiv']}}},\n archivePrefix = {{arXiv}},\n"
|
||||
f" note = {{arXiv preprint}},\n url = {{https://arxiv.org/abs/{rec['arxiv']}}}\n}}")
|
||||
|
||||
|
||||
def rekey_bibtex(entry, key):
|
||||
"""Replace the entry key (doi.org / dblp keys are ugly) and tidy whitespace."""
|
||||
entry = re.sub(r"^\s*@(\w+)\s*\{\s*[^,]*,", lambda m: f"@{m.group(1).lower()}{{{key},", entry.strip(), count=1)
|
||||
return re.sub(r"\n\s*\n", "\n", entry)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- network commands
|
||||
def search(query, max_n=25, since=None):
|
||||
q = urllib.parse.quote(query)
|
||||
cands = []
|
||||
atom = get(f"https://export.arxiv.org/api/query?search_query=all:{q}&max_results={max_n}&sortBy=relevance", "application/atom+xml")
|
||||
if atom:
|
||||
cands += parse_arxiv(atom)
|
||||
oa = get(f"https://api.openalex.org/works?search={q}&per-page={max_n}&mailto={urllib.parse.quote(UA.split('mailto:')[1][:-1])}")
|
||||
if oa:
|
||||
cands += parse_openalex(oa)
|
||||
time.sleep(1)
|
||||
s2 = get(f"https://api.semanticscholar.org/graph/v1/paper/search?query={q}&limit={max_n}"
|
||||
"&fields=title,authors,year,abstract,venue,publicationVenue,externalIds,citationCount,openAccessPdf,publicationTypes")
|
||||
if s2:
|
||||
cands += parse_s2(s2)
|
||||
merged = [c for c in merge(cands) if not since or (c.get("year") or "0") >= str(since)]
|
||||
merged.sort(key=lambda c: -(c.get("citations") or 0))
|
||||
for c in merged:
|
||||
c["key"] = bib_key(c.get("authors", []), c.get("year"), c.get("title"))
|
||||
Path("candidates.json").write_text(json.dumps(merged, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
print(f"{len(merged)} candidates (candidates.json). cites year venue key / title")
|
||||
for c in merged:
|
||||
venue = (c.get("venue") or ("arXiv" if c.get("arxiv") else "?"))[:26]
|
||||
print(f"{c.get('citations') or 0:6d} {c.get('year') or ' '} {venue:26s} {c['key']} {c['title'][:70]}")
|
||||
return merged
|
||||
|
||||
|
||||
def resolve(ident):
|
||||
"""Find the peer-reviewed version of a paper given an arXiv id, DOI or title."""
|
||||
arxiv = ARXIV_RE.fullmatch(ident.strip()) or (ARXIV_RE.search(ident) if "arxiv" in ident.lower() else None)
|
||||
doi = DOI_RE.search(ident)
|
||||
rec = None
|
||||
if arxiv or (doi and not is_arxiv_doi(doi.group(1))):
|
||||
pid = f"arXiv:{arxiv.group(1)}" if arxiv else doi.group(1)
|
||||
js = get(f"https://api.semanticscholar.org/graph/v1/paper/{pid}?fields=title,authors,year,venue,publicationVenue,externalIds,citationCount,openAccessPdf,publicationTypes")
|
||||
rec = parse_s2(js)[0] if js and not js.startswith('{"error"') else None
|
||||
title = rec["title"] if rec else ident.strip()
|
||||
result = {"query": ident, "title": title, "preprint": None, "published": None}
|
||||
if rec and rec.get("arxiv"):
|
||||
result["preprint"] = {"arxiv": rec["arxiv"], "url": f"https://arxiv.org/abs/{rec['arxiv']}", "pdf": f"https://arxiv.org/pdf/{rec['arxiv']}"}
|
||||
# 1. Semantic Scholar already knows a non-arXiv DOI + venue?
|
||||
if rec and rec.get("doi") and not is_arxiv_doi(rec["doi"]) and rec.get("venue"):
|
||||
result["published"] = {"doi": rec["doi"], "venue": rec["venue"], "year": rec["year"], "via": "semanticscholar",
|
||||
"type": rec.get("venue_type") or ",".join(rec.get("types", []))}
|
||||
time.sleep(1)
|
||||
# 2. Crossref by exact title (journal/proceedings only)
|
||||
if not result["published"]:
|
||||
cr = get(f"https://api.crossref.org/works?query.bibliographic={urllib.parse.quote(title)}&rows=5&mailto={urllib.parse.quote(UA.split('mailto:')[1][:-1])}")
|
||||
hit = parse_crossref(cr, title) if cr else None
|
||||
if hit:
|
||||
result["published"] = {"doi": hit["doi"], "venue": hit["venue"], "year": hit["year"], "via": "crossref", "type": hit["type"]}
|
||||
# 3. dblp (CS): venue + a clean .bib
|
||||
db = get(f"https://dblp.org/search/publ/api?q={urllib.parse.quote(title)}&format=json&h=10")
|
||||
hit = parse_dblp(db, title) if db else None
|
||||
if hit and hit["venue"] != "CoRR":
|
||||
pub = result["published"] or {}
|
||||
pub.update({k: v for k, v in {"venue": hit["venue"], "year": hit["year"], "doi": pub.get("doi") or hit.get("doi"),
|
||||
"dblp": hit["key"], "bib": hit["bib"], "type": hit["type"]}.items() if v})
|
||||
pub["via"] = pub.get("via") or "dblp"
|
||||
result["published"] = pub
|
||||
authors = rec["authors"] if rec else []
|
||||
result["key"] = bib_key(authors, (result["published"] or {}).get("year") or (rec or {}).get("year"), title)
|
||||
print(json.dumps(result, indent=2, ensure_ascii=False))
|
||||
if result["published"]:
|
||||
print(f"\n-> cite the published version: {result['published'].get('venue')} ({result['published'].get('year')}), "
|
||||
f"DOI {result['published'].get('doi')}; bibtex: python papers.py bibtex {result['published'].get('doi') or result['published'].get('dblp')}")
|
||||
elif result["preprint"]:
|
||||
print(f"\n-> no peer-reviewed version found: cite as arXiv preprint {result['preprint']['arxiv']} and say so in research.md")
|
||||
return result
|
||||
|
||||
|
||||
def bibtex(ident):
|
||||
ident = ident.strip()
|
||||
if ident.startswith("conf/") or ident.startswith("journals/"): # dblp key
|
||||
entry = get(f"https://dblp.org/rec/{ident}.bib", "text/plain")
|
||||
rec = None
|
||||
elif (m := DOI_RE.search(ident)) and not is_arxiv_doi(m.group(1)):
|
||||
entry = get(f"https://doi.org/{m.group(1)}", "application/x-bibtex")
|
||||
rec = None
|
||||
else:
|
||||
aid = ARXIV_RE.search(ident)
|
||||
if not aid:
|
||||
print("give a DOI, an arXiv id or a dblp key", file=sys.stderr); return None
|
||||
atom = get(f"https://export.arxiv.org/api/query?id_list={aid.group(1)}", "application/atom+xml")
|
||||
rec = parse_arxiv(atom)[0] if atom else None
|
||||
entry = arxiv_bibtex(rec) if rec else None
|
||||
if not entry:
|
||||
print("no BibTeX found", file=sys.stderr); return None
|
||||
m_auth = re.search(r"author\s*=\s*[{\"](.+?)[}\"]\s*,?\s*\n", entry, re.S | re.I)
|
||||
m_year = re.search(r"year\s*=\s*[{\"]?(\d{4})", entry, re.I)
|
||||
m_title = re.search(r"title\s*=\s*[{\"](.+?)[}\"]\s*,?\s*\n", entry, re.S | re.I)
|
||||
authors = [a.strip() for a in (m_auth.group(1) if m_auth else "").replace("\n", " ").split(" and ")]
|
||||
authors = [(" ".join(reversed(a.split(", "))) if ", " in a else a) for a in authors]
|
||||
key = bib_key(authors, m_year.group(1) if m_year else "", re.sub(r"[{}]", "", m_title.group(1)) if m_title else "")
|
||||
entry = rekey_bibtex(entry, key)
|
||||
print(entry)
|
||||
return entry
|
||||
|
||||
|
||||
def download(ident, out="papers"):
|
||||
Path(out).mkdir(parents=True, exist_ok=True)
|
||||
aid = ARXIV_RE.search(ident)
|
||||
url, name = None, None
|
||||
if ident.startswith("http"):
|
||||
url, name = ident, re.sub(r"[^\w.-]", "_", ident.rsplit("/", 1)[-1]) or "paper.pdf"
|
||||
elif aid:
|
||||
url, name = f"https://arxiv.org/pdf/{aid.group(1)}", f"arxiv_{aid.group(1).replace('.', '_')}.pdf"
|
||||
elif (m := DOI_RE.search(ident)):
|
||||
oa = get(f"https://api.openalex.org/works/https://doi.org/{m.group(1)}")
|
||||
best = (json.loads(oa).get("best_oa_location") or {}) if oa else {}
|
||||
url, name = best.get("pdf_url"), re.sub(r"[^\w.-]", "_", m.group(1)) + ".pdf"
|
||||
if not url:
|
||||
print("no open-access PDF known for this DOI (try the arXiv version, or the publisher page via the library)"); return None
|
||||
if not url:
|
||||
print("give an arXiv id, a DOI or a URL", file=sys.stderr); return None
|
||||
req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept": "application/pdf"})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=60) as r:
|
||||
data = r.read()
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"download failed: {e}", file=sys.stderr); return None
|
||||
if not data.startswith(b"%PDF"):
|
||||
print(f"not a PDF (got {data[:40]!r}) - probably a landing page or a captcha"); return None
|
||||
path = Path(out) / name
|
||||
path.write_bytes(data)
|
||||
print(f"saved {path} ({len(data) // 1024} KB)")
|
||||
return path
|
||||
|
||||
|
||||
def text(pdf, pages=None):
|
||||
try:
|
||||
import pymupdf as fitz
|
||||
except ImportError:
|
||||
try:
|
||||
import fitz
|
||||
except ImportError:
|
||||
print("pip install pymupdf", file=sys.stderr); return None
|
||||
doc = fitz.open(pdf)
|
||||
n = min(doc.page_count, pages or doc.page_count)
|
||||
parts = [f"\n\n===== page {i + 1} =====\n" + doc[i].get_text() for i in range(n)]
|
||||
out = Path(pdf).with_suffix(".txt")
|
||||
out.write_text("".join(parts), encoding="utf-8")
|
||||
print(f"{out}: {n}/{doc.page_count} pages, {sum(len(p) for p in parts)} chars")
|
||||
return out
|
||||
|
||||
|
||||
def main(argv):
|
||||
if not argv or argv[0] in ("-h", "--help"):
|
||||
print(__doc__); return 0
|
||||
cmd, args = argv[0], argv[1:]
|
||||
opt = {a.split("=")[0]: True for a in args if a.startswith("--")}
|
||||
def val(flag, default=None):
|
||||
if flag in args:
|
||||
i = args.index(flag); return args[i + 1]
|
||||
return default
|
||||
pos = [a for i, a in enumerate(args) if not a.startswith("--") and (i == 0 or not args[i - 1].startswith("--"))]
|
||||
if cmd == "search":
|
||||
search(" ".join(pos), int(val("--max", 25)), val("--since"))
|
||||
elif cmd == "resolve":
|
||||
resolve(" ".join(pos))
|
||||
elif cmd == "bibtex":
|
||||
bibtex(" ".join(pos))
|
||||
elif cmd == "download":
|
||||
download(pos[0], val("--out", "papers"))
|
||||
elif cmd == "text":
|
||||
text(pos[0], int(val("--pages")) if val("--pages") else None)
|
||||
else:
|
||||
print(__doc__); return 2
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
|
|
@ -0,0 +1,154 @@
|
|||
---
|
||||
name: single-file-webapp
|
||||
description: >
|
||||
Patterns for browser apps delivered as ONE self-contained HTML file (ES modules
|
||||
from a CDN import map, UI wiring, state, save/load JSON, file import/export,
|
||||
undo, test hooks, compact mobile-friendly layout) plus a static checker. Use for
|
||||
any "build a tool/editor/game in a single HTML file" task, before writing the first line.
|
||||
---
|
||||
|
||||
# Single-file web app
|
||||
|
||||
## Structure
|
||||
```html
|
||||
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><title>…</title>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
|
||||
<style>/* layout: CSS grid with named areas (menu / left / main / right) */</style>
|
||||
<script type="importmap">{ "imports": { "lib": "https://cdn.jsdelivr.net/npm/lib@X.Y.Z/..." } }</script>
|
||||
</head><body>
|
||||
<!-- markup with ids for everything JS touches -->
|
||||
<script type="module">
|
||||
import … from 'lib';
|
||||
const state = { objects: [], selection: new Set(), nextId: 1 };
|
||||
// --- helpers, model, render, UI wiring, test hook --- (in this order)
|
||||
</script></body></html>
|
||||
```
|
||||
Pin CDN versions (`@0.160.0`, never `@latest`). Keep everything in the file; no
|
||||
build step, no npm for the app itself.
|
||||
|
||||
## The five rules that prevent the usual bugs
|
||||
1. **No inline handlers with a module script.** `onclick="save()"` cannot see
|
||||
functions declared inside `<script type="module">` ("save is not defined").
|
||||
Wire every control with `document.getElementById('save').addEventListener('click', save)`.
|
||||
Assign `window.x = …` only for the test hook, and before anything uses it.
|
||||
2. **One state object, one render.** All data lives in `state`; `render()`
|
||||
redraws from it. Mutations go through small functions (`addObject`, `remove`,
|
||||
`group`) that end with `render()`. Never keep the truth in the DOM.
|
||||
3. **Serialize data, not live objects** (full recipe, checker and tests: the `save-load-json` skill - load it for any save/load/export feature). `toJSON(state)` returns plain objects
|
||||
(ids, type, params, transform, children for groups - recursively);
|
||||
`fromJSON(data)` rebuilds. Round-trip test: `fromJSON(toJSON(s))` equals `s`.
|
||||
Groups/nesting are the classic thing to forget - write them first.
|
||||
4. **Files:** export with `download(name, text)` (Blob + `<a download>`), import
|
||||
with `<input type="file">` + `FileReader`. Exporters are plain functions that
|
||||
RETURN the data (`toSTL()`, `toJSON()`) and the button handler is exactly
|
||||
`download('scene.json', JSON.stringify(toJSON(), null, 2))` - never a second,
|
||||
hand-written serializer in the handler. Importers likewise: the file handler
|
||||
only reads the file and calls `importSTL(buffer)` / `importSVG(text)`, which
|
||||
create objects through the normal constructor path. Test import/export via
|
||||
the buttons (`h.upload` / `h.download` in the browser-test skill).
|
||||
5. **Test hook** at the end of the module - the browser-test skill drives and
|
||||
asserts through it, so include at least:
|
||||
```js
|
||||
window.__app = {
|
||||
state, // live state object
|
||||
add: (type, params) => addObject(type, params), // returns the new object's id
|
||||
select: ids => { state.selection = new Set(ids); render(); updatePanel(); },
|
||||
project: id => { /* world position of object -> {x, y} page pixels */ },
|
||||
toJSON, fromJSON, toSTL, // pure data functions
|
||||
};
|
||||
```
|
||||
Tests set up state through `add`/`select`, use real clicks only to test
|
||||
clicking itself, and never rely on guessed pixel coordinates.
|
||||
|
||||
Reference snippets for all of the above (download, file input, undo via
|
||||
JSON snapshots, keyboard shortcuts, selection with shift, grid layout):
|
||||
[patterns.md](./patterns.md).
|
||||
|
||||
## Layout: compact, mobile first
|
||||
Design for a phone (~360-400 px wide) first, then let it grow on wide screens.
|
||||
The screen is small and the thumb is big: every pixel must earn its place.
|
||||
Snippets: "Mobile layout" in [patterns.md](./patterns.md).
|
||||
|
||||
**Menus and buttons**
|
||||
- **Symbols instead of words.** Toolbar buttons show an icon/emoji or a short
|
||||
symbol (💾 save, 📂 open, ↶ undo, ↷ redo, ＋ add, 🗑 delete, ⚙ settings, ☰ menu,
|
||||
⤢ fit/fullscreen); put the full name in `title` and `aria-label`. Use plain
|
||||
words only for the one or two primary actions, or when no clear symbol exists.
|
||||
- **Few top-level buttons.** More than ~6 actions: group them behind one `☰` /
|
||||
`⋯` button that opens a small dropdown. Never let the toolbar wrap onto a
|
||||
second row; make it one row that scrolls horizontally (`overflow-x:auto`).
|
||||
- **Touch targets** at least 40x40 px (`min-width/min-height:40px`), with 4-8 px
|
||||
gaps. Icon buttons are square, no wide padding.
|
||||
- Don't rely on hover (no tooltips-only info, no hover menus). Use `click`/`pointer` events.
|
||||
|
||||
**Input fields**
|
||||
- **Size fields to their content**, never full width by default: a number
|
||||
with up to 4 digits gets `width:5ch`, a hex colour `8ch`, a name `12ch`. Free
|
||||
text that really needs the width may flex. Set `type="number"` with
|
||||
`inputmode="decimal"` (or `numeric`) so phones show the number pad, and
|
||||
`step`/`min`/`max`.
|
||||
- **Group related fields on one row** (X Y Z, W H D, R G B) with a tiny
|
||||
label or symbol in front (`x`, `y`, `⌀`, `↔`, `↕`). Use `display:flex;gap:4px;flex-wrap:wrap`
|
||||
and a small row gap. No big margins between fields, no label above each field.
|
||||
- **Font size of inputs at least 16 px**, otherwise iOS zooms in on focus; make
|
||||
the fields compact via small padding (`2px 4px`), not small text.
|
||||
- Prefer sliders/steppers (`−` `+`) for values changed often, number fields
|
||||
for exact entry; checkboxes/toggles as `☑` symbol + short label.
|
||||
|
||||
**Panels, popups, dialogs**
|
||||
- **Side panels become drawers or bottom sheets on narrow screens**
|
||||
(`@media (max-width: 700px)`): hidden by default, opened with a symbol button,
|
||||
and they cover at most ~40 % of the height (bottom sheet) or ~70 % of the width
|
||||
(drawer) - the main area (canvas, board, preview) stays visible and usable.
|
||||
- **Popups must not hide what the user is working on.** Prefer a small dropdown
|
||||
anchored to its button, or a bottom sheet, over a centred modal. Limit
|
||||
`max-width:min(92vw, 360px)` and `max-height:50vh` (use `dvh`), always
|
||||
`overflow:auto` so content scrolls inside the popup, never the whole page.
|
||||
No full-screen modals except for real file/text editing.
|
||||
- Popups close on outside tap, `Escape` and via a visible `✕`; only one open at a time.
|
||||
Never use `alert()`/`confirm()`/`prompt()` - they block and look bad on phones;
|
||||
use a small inline message or toast (`position:fixed;bottom:8px`, auto-hide
|
||||
after 2-3 s) and don't let it cover the main controls.
|
||||
- Show results/status in a one-line status bar, not in popups.
|
||||
|
||||
**Page structure**
|
||||
- Use `100dvh` (fallback `100vh`) and `overflow:hidden` on `body` for app-like
|
||||
tools; scrolling happens inside panels, not on the page. The main area takes all
|
||||
remaining space (`1fr` / `flex:1; min-height:0`) and canvases are resized on
|
||||
`resize` and `orientationchange`.
|
||||
- Respect notches: `padding: env(safe-area-inset-top) ... ` on fixed bars.
|
||||
- Base font 14-16 px, line-height 1.2-1.3, tight spacing (4-8 px); use
|
||||
`rem`/`ch`/`%`/`min()`/`clamp()` rather than fixed pixel widths. No horizontal
|
||||
page scroll at 360 px width.
|
||||
- **Touch input** on canvas/board: use pointer events (`pointerdown/move/up`),
|
||||
`touch-action:none` on the canvas (otherwise the page scrolls instead of
|
||||
dragging), support two-finger pinch/pan for zoom where it matters, and
|
||||
keep a keyboard shortcut for each action as well (desktop).
|
||||
- Colours: sufficient contrast, don't encode meaning by colour alone; support
|
||||
`prefers-color-scheme` only if cheap.
|
||||
|
||||
## Games and simulations
|
||||
Rules 3 and 4 (files) usually don't apply; rules 1, 2 and 5 do, plus:
|
||||
- **Separate the logic from the clock.** `step()` advances the world by exactly
|
||||
one tick and never reads timers or `Date`; the loop is only
|
||||
`setInterval(() => { step(); render(); }, TICK_MS)` (or a `requestAnimationFrame`
|
||||
loop with an accumulator). That makes the game deterministic and testable.
|
||||
- **Input:** key handlers only record the wanted action in a variable that
|
||||
`step()` reads. Nothing moves inside a key handler.
|
||||
- **Game over** is state (`state.over = 'won' | 'lost' | null`), not control
|
||||
flow: `step()` returns early when set, `render()` shows it, restart rebuilds
|
||||
the state from scratch.
|
||||
- **Test hook:** `window.__app = { get state(), step, setInput, reset }` - tests
|
||||
call `__app.step()` in a loop instead of waiting for ticks.
|
||||
|
||||
## Verify
|
||||
1. `python <skill-dir>/check_html.py app.html` - syntax of every script block,
|
||||
import map JSON, inline handlers vs. module scope, missing element ids,
|
||||
mobile basics (viewport meta, `alert/confirm/prompt`, tiny input font sizes,
|
||||
fixed pixel widths on inputs, popups without `max-height`/`overflow`).
|
||||
2. Run it in the headless browser with the `browser-test` skill (errors, UI
|
||||
actions, screenshots) - ALSO at phone size (viewport 390x844, and landscape
|
||||
844x390): take a screenshot, check that the toolbar is a single row, nothing
|
||||
is cut off, there is no horizontal scrollbar
|
||||
(`document.documentElement.scrollWidth <= innerWidth`), and that an opened
|
||||
popup/panel leaves the main area visible.
|
||||
|
|
@ -0,0 +1,85 @@
|
|||
#!/usr/bin/env python3
|
||||
"""check_html.py - static smoke test for a single-file web app.
|
||||
|
||||
python check_html.py app.html
|
||||
|
||||
Checks (exit code 1 on any finding):
|
||||
- every <script> block parses (node --check; module scripts as .mjs)
|
||||
- <script type="importmap"> is valid JSON with an "imports" object
|
||||
- inline handlers (onclick="...") used together with a module script: they
|
||||
cannot see module-scope functions -> use addEventListener inside the module
|
||||
- ids referenced via getElementById('x') / querySelector('#x') exist in the HTML
|
||||
- mobile basics: viewport meta, no alert/confirm/prompt, input font-size >= 16px,
|
||||
no fixed pixel widths >= 100px on inputs, popups/modals with max-height + overflow:auto
|
||||
"""
|
||||
import json, re, subprocess, sys, tempfile
|
||||
from pathlib import Path
|
||||
|
||||
def main(path: str) -> int:
|
||||
html = Path(path).read_text(encoding="utf-8")
|
||||
findings = []
|
||||
scripts = re.findall(r"<script\b([^>]*)>(.*?)</script>", html, re.S | re.I)
|
||||
has_module = any('type="module"' in a or "type='module'" in a for a, _ in scripts)
|
||||
|
||||
for i, (attrs, body) in enumerate(scripts, 1):
|
||||
if "importmap" in attrs:
|
||||
try:
|
||||
im = json.loads(body)
|
||||
if not isinstance(im.get("imports"), dict):
|
||||
findings.append(f"script {i}: importmap has no 'imports' object")
|
||||
except json.JSONDecodeError as e:
|
||||
findings.append(f"script {i}: importmap is not valid JSON ({e})")
|
||||
continue
|
||||
if "src=" in attrs or not body.strip():
|
||||
continue
|
||||
suffix = ".mjs" if "module" in attrs else ".js"
|
||||
with tempfile.NamedTemporaryFile("w", suffix=suffix, delete=False, encoding="utf-8") as f:
|
||||
f.write(body); tmp = f.name
|
||||
r = subprocess.run(["node", "--check", tmp], capture_output=True, text=True)
|
||||
if r.returncode:
|
||||
msg = r.stderr.strip().splitlines()
|
||||
findings.append(f"script {i}: syntax error - " + " | ".join(l for l in msg if l.strip())[:300])
|
||||
|
||||
if has_module:
|
||||
for m in re.finditer(r"\son(\w+)\s*=\s*[\"']([^\"']+)[\"']", html):
|
||||
findings.append(f"inline on{m.group(1)}=\"{m.group(2)[:40]}\" with a module script "
|
||||
"- handler can't reach module scope; use addEventListener")
|
||||
|
||||
ids = set(re.findall(r"\bid\s*=\s*[\"']([^\"']+)[\"']", html))
|
||||
used = set(re.findall(r"getElementById\(\s*[\"']([^\"']+)[\"']", html))
|
||||
used |= set(re.findall(r"querySelector(?:All)?\(\s*[\"']#([A-Za-z_][\w-]*)[\"']", html))
|
||||
for u in sorted(used - ids):
|
||||
findings.append(f"element id '{u}' is referenced in JS but does not exist in the HTML")
|
||||
|
||||
# --- mobile basics -------------------------------------------------------
|
||||
if not re.search(r"<meta[^>]+name=[\"']viewport[\"'][^>]*width=device-width", html, re.I):
|
||||
findings.append('missing <meta name="viewport" content="width=device-width, initial-scale=1"> '
|
||||
"- the page renders as a tiny desktop page on phones")
|
||||
for m in re.finditer(r"\b(alert|confirm|prompt)\s*\(", "\n".join(b for a, b in scripts if "src=" not in a)):
|
||||
findings.append(f"{m.group(1)}() blocks and looks bad on phones - use a toast / small popup instead")
|
||||
css = "\n".join(re.findall(r"<style\b[^>]*>(.*?)</style>", html, re.S | re.I))
|
||||
for m in re.finditer(r"(?:^|})\s*([^{}]*\b(?:input|select|textarea)\b[^{}]*)\{([^}]*)\}", css):
|
||||
fs = re.search(r"font-size\s*:\s*(\d+(?:\.\d+)?)px", m.group(2))
|
||||
if fs and float(fs.group(1)) < 16:
|
||||
findings.append(f"font-size {fs.group(1)}px on '{m.group(1).strip()[:40]}' - below 16px iOS zooms in on focus; "
|
||||
"keep 16px and make fields compact with padding/width")
|
||||
for m in re.finditer(r"(?:^|})\s*([^{}]*\b(?:input|select)\b[^{}]*)\{([^}]*)\}", css):
|
||||
if re.search(r"(?<![-\w])width\s*:\s*(\d{3,})px", m.group(2)):
|
||||
findings.append(f"fixed pixel width >=100px on '{m.group(1).strip()[:40]}' - size fields to their "
|
||||
"content with ch units (e.g. width:5ch) or max-width")
|
||||
for m in re.finditer(r"(?:^|})\s*([^{}]*(?:popup|modal|dialog|sheet|overlay)[^{}]*)\{([^}]*)\}", css, re.I):
|
||||
body = m.group(2)
|
||||
if re.search(r"position\s*:\s*(fixed|absolute)", body) and not (
|
||||
re.search(r"max-height", body) and re.search(r"overflow(-y)?\s*:\s*(auto|scroll)", body)):
|
||||
findings.append(f"popup-like '{m.group(1).strip()[:40]}' is positioned but has no max-height + overflow:auto "
|
||||
"- it can cover the screen and cannot scroll")
|
||||
|
||||
for f_ in findings:
|
||||
print("✗", f_)
|
||||
print("ok" if not findings else f"{len(findings)} finding(s)")
|
||||
return 1 if findings else 0
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
print(__doc__); sys.exit(2)
|
||||
sys.exit(main(sys.argv[1]))
|
||||
|
|
@ -0,0 +1,164 @@
|
|||
# Snippets for single-file web apps
|
||||
|
||||
## Layout (menu on top, panels left/right, main area fills)
|
||||
```css
|
||||
html,body{margin:0;height:100%;font:14px system-ui}
|
||||
body{display:grid;grid-template-rows:40px 1fr;grid-template-columns:200px 1fr 240px;
|
||||
grid-template-areas:"menu menu menu" "left main right"}
|
||||
#menu{grid-area:menu} #left{grid-area:left;overflow:auto} #main{grid-area:main;position:relative}
|
||||
#right{grid-area:right;overflow:auto} canvas{display:block}
|
||||
```
|
||||
Size a canvas/renderer from `#main.clientWidth/clientHeight`, and again on `resize`.
|
||||
|
||||
## Mobile layout (compact toolbar, side panels as bottom sheet, small popups)
|
||||
```html
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
|
||||
<div id="menu">
|
||||
<button id="open" title="Open" aria-label="Open">📂</button>
|
||||
<button id="save" title="Save" aria-label="Save">💾</button>
|
||||
<button id="undo" title="Undo" aria-label="Undo">↶</button>
|
||||
<button id="redo" title="Redo" aria-label="Redo">↷</button>
|
||||
<button id="add" title="Add" aria-label="Add">＋</button>
|
||||
<button id="more" title="More" aria-label="More">⋯</button>
|
||||
<span id="status"></span>
|
||||
</div>
|
||||
<div class="row"><label>x<input id="px" type="number" inputmode="decimal" class="n4"></label>
|
||||
<label>y<input id="py" type="number" inputmode="decimal" class="n4"></label>
|
||||
<label>z<input id="pz" type="number" inputmode="decimal" class="n4"></label></div>
|
||||
```
|
||||
```css
|
||||
:root{--gap:4px;--tap:40px}
|
||||
*{box-sizing:border-box}
|
||||
html,body{margin:0;height:100%;overflow:hidden;font:14px/1.25 system-ui;touch-action:manipulation}
|
||||
body{height:100dvh;display:grid;grid-template-rows:auto 1fr;grid-template-columns:1fr;
|
||||
grid-template-areas:"menu" "main"}
|
||||
/* one-row toolbar: icon buttons, scrolls sideways instead of wrapping */
|
||||
#menu{grid-area:menu;display:flex;gap:var(--gap);padding:calc(var(--gap)) var(--gap);
|
||||
padding-top:max(var(--gap),env(safe-area-inset-top));overflow-x:auto;white-space:nowrap;align-items:center}
|
||||
button{min-width:var(--tap);min-height:var(--tap);padding:0 8px;font-size:18px;line-height:1}
|
||||
#status{margin-left:auto;font-size:12px;opacity:.7}
|
||||
#main{grid-area:main;position:relative;min-height:0} canvas{display:block;touch-action:none}
|
||||
|
||||
/* compact fields: width fits the content, text >=16px (no iOS zoom), small padding */
|
||||
input,select{font-size:16px;padding:2px 4px;min-height:32px}
|
||||
.n4{width:5ch} .n6{width:7ch} .name{width:12ch} .hex{width:8ch}
|
||||
.row{display:flex;flex-wrap:wrap;gap:var(--gap) 8px;align-items:center;margin:2px 0}
|
||||
.row label{display:flex;align-items:center;gap:2px} /* tiny label left of the field */
|
||||
|
||||
/* side panels: drawer / bottom sheet on phones, fixed columns on wide screens */
|
||||
#left,#right{background:Canvas;overflow:auto;padding:var(--gap)}
|
||||
@media (max-width:700px){
|
||||
#left,#right{position:fixed;left:0;right:0;bottom:0;max-height:40dvh;z-index:5;
|
||||
border-top:1px solid #8886;transform:translateY(100%);transition:transform .15s}
|
||||
#left.open,#right.open{transform:none} /* toggled by a ☰ / ⚙ button; main area stays visible above */
|
||||
}
|
||||
@media (min-width:701px){
|
||||
body{grid-template-columns:200px 1fr 240px;grid-template-areas:"menu menu menu" "left main right"}
|
||||
#left{grid-area:left} #right{grid-area:right}
|
||||
}
|
||||
|
||||
/* popups: small, anchored, scrollable inside, never full-screen */
|
||||
.popup{position:fixed;z-index:10;max-width:min(92vw,360px);max-height:50dvh;overflow:auto;
|
||||
background:Canvas;color:CanvasText;border:1px solid #8886;border-radius:6px;padding:6px;
|
||||
box-shadow:0 4px 16px #0004}
|
||||
.popup[hidden]{display:none}
|
||||
.toast{position:fixed;left:50%;bottom:8px;transform:translateX(-50%);max-width:90vw;
|
||||
background:#222;color:#fff;padding:4px 10px;border-radius:12px;font-size:13px;z-index:20;pointer-events:none}
|
||||
```
|
||||
```js
|
||||
// popup anchored below its button, kept inside the viewport; closes on outside tap / Escape
|
||||
function openPopup(pop, anchor) {
|
||||
document.querySelectorAll('.popup').forEach(p => p.hidden = true); // only one open
|
||||
pop.hidden = false;
|
||||
const a = anchor.getBoundingClientRect(), w = pop.offsetWidth, h = pop.offsetHeight;
|
||||
pop.style.left = Math.max(4, Math.min(a.left, innerWidth - w - 4)) + 'px';
|
||||
pop.style.top = (a.bottom + 4 + h > innerHeight ? Math.max(4, a.top - h - 4) : a.bottom + 4) + 'px';
|
||||
}
|
||||
document.addEventListener('pointerdown', e => {
|
||||
if (!e.target.closest('.popup, #menu button')) document.querySelectorAll('.popup').forEach(p => p.hidden = true);
|
||||
});
|
||||
document.addEventListener('keydown', e => { if (e.key === 'Escape') document.querySelectorAll('.popup').forEach(p => p.hidden = true); });
|
||||
function toast(msg, ms = 2500) { // instead of alert()
|
||||
const t = Object.assign(document.createElement('div'), { className: 'toast', textContent: msg });
|
||||
document.body.append(t); setTimeout(() => t.remove(), ms);
|
||||
}
|
||||
```
|
||||
Canvas: `canvas.style.touchAction = 'none'` and handle `pointerdown/move/up`
|
||||
(+ two-pointer pinch) so dragging does not scroll the page. Resize on `resize`
|
||||
and `orientationchange`.
|
||||
|
||||
## UI wiring
|
||||
```js
|
||||
const $ = id => document.getElementById(id);
|
||||
$('save').addEventListener('click', () => download('scene.json', JSON.stringify(toJSON(), null, 2)));
|
||||
$('posX').addEventListener('input', e => { if (sel) { sel.position.x = +e.target.value; render(); } });
|
||||
document.addEventListener('keydown', e => {
|
||||
if (e.target.tagName === 'INPUT') return; // don't steal keys from inputs
|
||||
if (e.key === 'Delete') removeSelected();
|
||||
if ((e.ctrlKey || e.metaKey) && e.key === 'z') undo();
|
||||
});
|
||||
```
|
||||
|
||||
## Download / file import
|
||||
```js
|
||||
function download(name, data, type = 'text/plain') {
|
||||
const a = document.createElement('a');
|
||||
a.href = URL.createObjectURL(new Blob([data], { type }));
|
||||
a.download = name; a.click(); URL.revokeObjectURL(a.href);
|
||||
}
|
||||
function pickFile(accept, asText, cb) { // asText=false -> ArrayBuffer (binary STL)
|
||||
const inp = document.createElement('input'); inp.type = 'file'; inp.accept = accept;
|
||||
inp.onchange = () => { const r = new FileReader();
|
||||
r.onload = () => cb(r.result, inp.files[0].name);
|
||||
asText ? r.readAsText(inp.files[0]) : r.readAsArrayBuffer(inp.files[0]); };
|
||||
inp.click();
|
||||
}
|
||||
```
|
||||
|
||||
## Serialization with nested groups
|
||||
```js
|
||||
function objToJSON(o) {
|
||||
return { id: o.id, type: o.type, params: { ...o.params }, hole: !!o.hole, color: o.color,
|
||||
position: [...o.position], rotation: [...o.rotation],
|
||||
children: o.type === 'group' ? o.children.map(objToJSON) : undefined };
|
||||
}
|
||||
const toJSON = () => ({ version: 1, grid: state.grid, objects: state.objects.map(objToJSON) });
|
||||
const fromJSON = d => { state.objects = d.objects.map(objFromJSON); state.grid = d.grid ?? 1; state.selection.clear(); render(); };
|
||||
```
|
||||
`objFromJSON` is the only place that creates runtime objects from data - the
|
||||
"add shape" buttons should call it too (`objFromJSON({type:'box', params:{...}})`),
|
||||
so there is exactly one constructor path.
|
||||
|
||||
## Undo / redo via snapshots
|
||||
```js
|
||||
const history = [], future = [];
|
||||
function commit() { history.push(JSON.stringify(toJSON())); future.length = 0; if (history.length > 50) history.shift(); }
|
||||
function undo() { if (!history.length) return; future.push(JSON.stringify(toJSON())); fromJSON(JSON.parse(history.pop())); }
|
||||
```
|
||||
Call `commit()` BEFORE each mutation.
|
||||
|
||||
## Selection with shift-click (bounding boxes drawn by render)
|
||||
```js
|
||||
function pick(ev) { /* hit test -> object or null */ }
|
||||
canvas.addEventListener('pointerdown', ev => {
|
||||
const o = pick(ev);
|
||||
if (!ev.shiftKey) state.selection.clear();
|
||||
if (o) state.selection.has(o) ? state.selection.delete(o) : state.selection.add(o);
|
||||
render(); updatePanel();
|
||||
});
|
||||
```
|
||||
|
||||
## Test hook (last lines of the module)
|
||||
```js
|
||||
window.__app = {
|
||||
state, toJSON, fromJSON, toSTL,
|
||||
add: (type, params = {}) => { const o = objFromJSON({ type, params }); state.objects.push(o); render(); return o.id; },
|
||||
select: ids => { state.selection = new Set(ids); render(); updatePanel(); },
|
||||
project: id => { // three.js: world position -> page pixels (for click tests)
|
||||
const o = state.objects.find(x => x.id === id);
|
||||
const v = new THREE.Vector3(...o.position).project(camera);
|
||||
const r = renderer.domElement.getBoundingClientRect();
|
||||
return { x: r.left + (v.x + 1) / 2 * r.width, y: r.top + (1 - v.y) / 2 * r.height };
|
||||
},
|
||||
};
|
||||
```
|
||||
|
|
@ -0,0 +1,84 @@
|
|||
---
|
||||
name: speech-synthesis
|
||||
description: >
|
||||
Text-to-speech with Kokoro-82M (open weights, runs on CPU): synthesize WAV
|
||||
files with a female or male voice, choose British/American English and other
|
||||
languages, blend voices, set the speed, batch long texts, and play or embed
|
||||
the result. Use for any TTS, voice-over, audiobook, speaking assistant or
|
||||
"read this aloud" task.
|
||||
---
|
||||
|
||||
# Speech synthesis with Kokoro
|
||||
|
||||
Kokoro-82M is a small, fast TTS model (Apache 2.0; ~330 MB download once).
|
||||
Use the ready-made script [tts.py](./tts.py) - copy it into the project
|
||||
(`cp <skill-dir>/tts.py .`) or import `synthesize()` from it.
|
||||
|
||||
## Setup
|
||||
```
|
||||
pip install kokoro soundfile # torch comes with it; first run downloads the weights
|
||||
apt install espeak-ng / brew install espeak-ng # only for non-English languages
|
||||
```
|
||||
Python 3.10-3.12 recommended. CPU is fine: ~1 s of audio per 0.1-0.3 s compute.
|
||||
A GPU is used automatically if torch sees one.
|
||||
|
||||
## Female and male voice - the two calls that matter
|
||||
```python
|
||||
from kokoro import KPipeline
|
||||
import soundfile as sf, numpy as np
|
||||
|
||||
pipeline = KPipeline(lang_code='a') # 'a' American English, 'b' British English
|
||||
|
||||
# feminine voice
|
||||
audio = np.concatenate([a for _, _, a in pipeline("Good morning! The report is ready.", voice='af_heart')])
|
||||
sf.write('female.wav', audio, 24000)
|
||||
|
||||
# masculine voice
|
||||
audio = np.concatenate([a for _, _, a in pipeline("Good morning! The report is ready.", voice='am_michael')])
|
||||
sf.write('male.wav', audio, 24000)
|
||||
```
|
||||
`pipeline(text, voice=..., speed=1.0)` is a generator: one `(graphemes,
|
||||
phonemes, audio)` item per chunk (chunks split at newlines by default). Audio
|
||||
is a float32 array at **24 kHz**. Voices: `af_*`/`am_*` American female/male,
|
||||
`bf_*`/`bm_*` British. Good defaults: `af_heart` (F), `am_michael` (M),
|
||||
`bf_emma` (F, British), `bm_george` (M, British). `python tts.py --list-voices`
|
||||
prints all.
|
||||
|
||||
## Script
|
||||
```
|
||||
python tts.py "Hello there." --out hello.wav # af_heart (female)
|
||||
python tts.py "Hello there." --voice am_michael --out hello_m.wav
|
||||
python tts.py --file chapter.txt --voice bf_emma --lang b --speed 0.9 --out chapter.wav
|
||||
python tts.py "Blend." --voice af_heart,am_michael --out blend.wav # average of two voices
|
||||
```
|
||||
`synthesize(text, voice, lang, speed) -> (samples, 24000)` splits at blank
|
||||
lines and sentence ends, inserts 250 ms pauses and concatenates - no length limit.
|
||||
|
||||
## Languages
|
||||
Kokoro-82M v1.0: `a` American English, `b` British English, `e` Spanish,
|
||||
`f` French, `h` Hindi, `i` Italian, `j` Japanese, `p` Brazilian Portuguese,
|
||||
`z` Mandarin (`pip install misaki[ja]` / `misaki[zh]` for j/z). **No German.**
|
||||
For German use Piper (`pip install piper-tts`, voice `de_DE-thorsten-medium`)
|
||||
and say so in the summary instead of feeding German text to an English voice.
|
||||
|
||||
## Using the audio
|
||||
- Play: `pip install sounddevice` then `sd.play(samples, 24000); sd.wait()`,
|
||||
or `python tts.py ... --play`.
|
||||
- MP3/OGG: write WAV, convert with `ffmpeg -i out.wav -b:a 96k out.mp3`.
|
||||
- Web page: serve the WAV and use `<audio controls src="out.wav">`; for an
|
||||
in-browser TTS without Python see kokoro-js (`kokoro-js` npm, WebGPU/WASM).
|
||||
- Verify: check the duration (`len(samples)/24000`) is plausible for the text
|
||||
(~2.5 words/s) and that `np.abs(samples).max() > 0.05` (not silence); for a
|
||||
listening check write a short sample and tell the user the file name.
|
||||
|
||||
## Pitfalls
|
||||
- First run is slow (weight download + model load); keep ONE `KPipeline` per
|
||||
language and reuse it (`tts.py` caches it).
|
||||
- Text with no sentence punctuation is spoken as one breathless chunk - add
|
||||
periods or newlines.
|
||||
- Numbers, dates and abbreviations are read as written; expand them in the
|
||||
text ("3 km" -> "three kilometres") for clean output.
|
||||
- Empty audio / `None` in the generator: the chunk was empty or unpronounceable
|
||||
(only symbols). Skip it, don't crash.
|
||||
- `numpy` must be < 2.0 for current kokoro releases (`pip install "numpy<2"` if
|
||||
the import fails).
|
||||
|
|
@ -0,0 +1,83 @@
|
|||
#!/usr/bin/env python3
|
||||
"""tts.py - text to speech with Kokoro-82M (open weights, Apache 2.0).
|
||||
|
||||
python tts.py "Hello, this is a test." --out hello.wav # female, American English
|
||||
python tts.py "Hello." --voice am_michael --out hello_m.wav # male
|
||||
python tts.py --file chapter.txt --voice bf_emma --lang b --out chapter.wav
|
||||
python tts.py "Mixed." --voice af_heart,am_michael --out mix.wav # blend of two voices
|
||||
python tts.py --list-voices
|
||||
|
||||
Setup (once): pip install kokoro soundfile (downloads ~330 MB of weights on first run)
|
||||
Languages other than English need: apt/brew install espeak-ng
|
||||
Output: 24 kHz mono 16-bit WAV. Long texts are split at blank lines / sentences,
|
||||
synthesized chunk by chunk and concatenated, so there is no length limit.
|
||||
"""
|
||||
import argparse, sys, time
|
||||
|
||||
# Kokoro-82M v1.0 voices (prefix: a/b = American/British English, f/m = female/male)
|
||||
VOICES = {
|
||||
"female American": ["af_heart", "af_bella", "af_nicole", "af_sarah", "af_sky", "af_aoede", "af_kore", "af_nova", "af_river", "af_jessica", "af_alloy"],
|
||||
"male American": ["am_michael", "am_adam", "am_fenrir", "am_puck", "am_echo", "am_eric", "am_liam", "am_onyx", "am_santa"],
|
||||
"female British": ["bf_emma", "bf_isabella", "bf_alice", "bf_lily"],
|
||||
"male British": ["bm_george", "bm_lewis", "bm_daniel", "bm_fable"],
|
||||
"other languages": ["ef_dora em_alex (es)", "ff_siwis (fr)", "hf_alpha hm_omega (hi)", "if_sara im_nicola (it)",
|
||||
"jf_alpha jm_kumo (ja)", "pf_dora pm_alex (pt-br)", "zf_xiaobei zm_yunjian (zh)"],
|
||||
}
|
||||
LANGS = {"a": "American English", "b": "British English", "e": "Spanish", "f": "French", "h": "Hindi",
|
||||
"i": "Italian", "j": "Japanese", "p": "Portuguese (BR)", "z": "Mandarin"} # no German in Kokoro-82M v1.0
|
||||
|
||||
|
||||
def synthesize(text: str, voice: str = "af_heart", lang: str = "a", speed: float = 1.0):
|
||||
"""Returns (samples: float32 numpy array, sample_rate=24000). Loads the model on first call."""
|
||||
import numpy as np
|
||||
from kokoro import KPipeline # heavy import (torch); keep it inside the function
|
||||
global _PIPELINES
|
||||
try:
|
||||
_PIPELINES
|
||||
except NameError:
|
||||
_PIPELINES = {}
|
||||
if lang not in _PIPELINES:
|
||||
_PIPELINES[lang] = KPipeline(lang_code=lang)
|
||||
chunks = []
|
||||
# the pipeline yields (graphemes, phonemes, audio) per text chunk; split_pattern decides the chunks
|
||||
for _graphemes, _phonemes, audio in _PIPELINES[lang](text, voice=voice, speed=speed, split_pattern=r"\n+|(?<=[.!?])\s+"):
|
||||
if audio is not None:
|
||||
chunks.append(audio.numpy() if hasattr(audio, "numpy") else np.asarray(audio))
|
||||
chunks.append(np.zeros(int(24000 * 0.25), dtype=np.float32)) # 250 ms pause between chunks
|
||||
if not chunks:
|
||||
raise ValueError("no audio produced - empty text?")
|
||||
return np.concatenate(chunks).astype(np.float32), 24000
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("text", nargs="?", help="text to speak (or use --file)")
|
||||
ap.add_argument("--file", help="read the text from this file")
|
||||
ap.add_argument("--voice", default="af_heart", help="voice name or 'name1,name2' blend (default af_heart = female)")
|
||||
ap.add_argument("--lang", default="a", choices=LANGS, help="language/G2P: a=American, b=British English, ...")
|
||||
ap.add_argument("--speed", type=float, default=1.0, help="0.5 .. 2.0")
|
||||
ap.add_argument("--out", default="out.wav", help="WAV file to write")
|
||||
ap.add_argument("--play", action="store_true", help="also play it (needs 'pip install sounddevice')")
|
||||
ap.add_argument("--list-voices", action="store_true")
|
||||
a = ap.parse_args()
|
||||
if a.list_voices:
|
||||
for group, names in VOICES.items():
|
||||
print(f"{group:17s} {' '.join(names)}")
|
||||
print("languages:", ", ".join(f"{k}={v}" for k, v in LANGS.items()))
|
||||
return 0
|
||||
text = open(a.file, encoding="utf-8").read() if a.file else a.text
|
||||
if not text or not text.strip():
|
||||
ap.error("give a text or --file")
|
||||
t0 = time.time()
|
||||
samples, sr = synthesize(text, a.voice, a.lang, a.speed)
|
||||
import soundfile as sf
|
||||
sf.write(a.out, samples, sr)
|
||||
print(f"wrote {a.out}: {len(samples) / sr:.1f} s of audio, voice {a.voice}, {time.time() - t0:.1f} s")
|
||||
if a.play:
|
||||
import sounddevice as sd
|
||||
sd.play(samples, sr); sd.wait()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -0,0 +1,112 @@
|
|||
---
|
||||
name: super-intelligence-write-skill
|
||||
description: >
|
||||
Thoughtful, research-backed writing in the voice of an eloquent professor. Before writing it asks
|
||||
a few open-ended questions about the topic, then researches the answers and the topic, and writes
|
||||
a precise, well-argued text with sources. Use for "write about", "explain in depth", "write an
|
||||
essay / article / statement / lecture text", or when the user wants a carefully reasoned, polished answer.
|
||||
---
|
||||
|
||||
# Super-intelligence write
|
||||
|
||||
Three phases, in this order: **Ask -> Research -> Write**. Never skip straight to writing.
|
||||
Reply in the language of the user's request.
|
||||
|
||||
## Phase 1 - Ask (then stop and wait)
|
||||
Before any research or writing, ask the user **3-5 open-ended questions**, numbered, ordered
|
||||
from most to least important. Then end your turn without tool calls and wait for the answers.
|
||||
|
||||
Good questions:
|
||||
- are open ("What should the reader be able to do after reading?") - never yes/no;
|
||||
- show that you have already thought about the topic: name the real tension or the choice the
|
||||
text must make ("Should the text explain *how* X works or argue *whether* X is desirable -
|
||||
these lead to very different pieces?");
|
||||
- each fit on one or two lines, with a short clause on why you ask when that isn't obvious.
|
||||
|
||||
Cover, as far as the request leaves it open:
|
||||
1. **Purpose and reader** - who reads this, what they already know, what they should think or
|
||||
do afterwards.
|
||||
2. **The core question** - which aspect, tension or thesis matters most to the user.
|
||||
3. **The user's own view and knowledge** - what they believe, have experienced, or have read;
|
||||
which sources or examples they want included or excluded.
|
||||
4. **Scope and form** - length, genre (essay, explainer, speech, statement, letter),
|
||||
register, deadline-relevant constraints.
|
||||
5. **What would make it excellent** in their eyes - or what they dislike in texts on this topic.
|
||||
|
||||
Skip questions the request already answers. If the user says "just write", has already given
|
||||
detailed answers, or no one can answer (unattended run), state in one line the assumptions you
|
||||
make for each open point and continue with Phase 2.
|
||||
|
||||
## Phase 2 - Research
|
||||
Research the topic **and the user's answers** (verify the facts they gave; follow up on the
|
||||
sources and examples they named).
|
||||
1. Write down 3-6 sub-questions the text must answer.
|
||||
2. Search the web for each (`web_search`, then read the best hits with `fetch_url`; prefer
|
||||
primary sources, reference works, peer-reviewed papers, official statistics, and recent
|
||||
sources for anything that changes over time). For a scientific topic, load the
|
||||
`scientific-deep-research` skill and use its procedure for papers and BibTeX.
|
||||
3. Keep notes (in `<slug>-notes.md` if you can write files): for every fact, the source URL and
|
||||
the exact figure or wording. Note where sources disagree and why.
|
||||
4. Find the strongest counter-position to the thesis the text will take, and the best evidence
|
||||
for it.
|
||||
5. Stop when new searches return what you already know. Depth beats breadth: five well-read
|
||||
sources beat twenty skimmed ones.
|
||||
Never present a fact you did not find or cannot attribute. If something cannot be verified,
|
||||
say so in the text ("the evidence here is thin") or leave it out.
|
||||
|
||||
## Phase 3 - Write like an eloquent professor
|
||||
The model is the best lecturer you ever had: someone who has thought long about the subject,
|
||||
respects the reader's intelligence, and makes hard things clear without making them simple.
|
||||
|
||||
### Thinking before writing
|
||||
- Formulate the **thesis in one sentence**. If you can't, the thinking isn't finished.
|
||||
- Outline the argument: claim -> distinctions -> evidence -> strongest objection -> answer ->
|
||||
synthesis. Every paragraph has one job.
|
||||
|
||||
### Precision
|
||||
- **Define the key terms** at first use and keep them fixed; don't switch synonyms for variety
|
||||
when the meaning must stay the same.
|
||||
- **Draw distinctions** that dissolve confusion ("Two questions are often conflated here: ...").
|
||||
- **Calibrate certainty**: "is", "is likely", "may", "is unknown" mean different things - use
|
||||
the one the evidence supports. Quantify where possible ("in roughly 3 of 4 trials", not "often").
|
||||
- Concrete over abstract: a well-chosen example, number or case after each general claim.
|
||||
- Name causes and mechanisms, not just correlations; say what would change your mind.
|
||||
|
||||
### Eloquence
|
||||
- Clarity first, elegance second, ornament never. Short sentences for claims; longer ones for
|
||||
reasoning; vary the rhythm.
|
||||
- One vivid analogy or image where it carries understanding - not as decoration.
|
||||
- Strong verbs, few adjectives, no filler ("it is important to note that", "in today's world",
|
||||
"delve", "tapestry", "landscape", "crucial", "game-changer").
|
||||
- Open with something that earns attention: a question, a surprising fact, a precise tension -
|
||||
never a dictionary definition or a platitude.
|
||||
- Close with synthesis: what follows, what remains open, why it matters. No mere summary.
|
||||
- Address objections fairly, in their strongest form, before answering them.
|
||||
- Intellectual humility without hedging everything: be clear where the evidence is clear.
|
||||
|
||||
### Form
|
||||
- Prose by default; headings for texts over ~800 words; lists only for genuinely list-like
|
||||
content.
|
||||
- Attribute claims in the text ("as Kahneman showed ...", "according to Eurostat 2025 ...") and
|
||||
end with a **Sources** list (author/title, year, URL). Academic texts: citations in the style
|
||||
the user wants, otherwise author-year.
|
||||
- Respect the requested length to within ~10%.
|
||||
- Texts longer than ~400 words: if you can write files, save them as `<slug>.md` and give a
|
||||
short summary in the reply.
|
||||
|
||||
## Phase 4 - Revise (always)
|
||||
Reread the draft as a demanding colleague and fix before delivering:
|
||||
- Is the thesis clear in the first paragraph? Does every paragraph serve it?
|
||||
- Is every factual claim backed by the notes? Every number checked against its source?
|
||||
- Any term used in two meanings? Any vague word ("many", "significant", "various") that can be
|
||||
made precise?
|
||||
- Is the strongest objection addressed honestly?
|
||||
- Cut 10%: redundant sentences, throat-clearing, repeated points.
|
||||
- Read the first sentence of each paragraph alone: do they tell the story?
|
||||
|
||||
## Example of the register
|
||||
Weak: "Artificial intelligence is a very important topic in today's world and has many
|
||||
advantages and disadvantages."
|
||||
Strong: "Whether artificial intelligence helps or harms is the wrong question; the useful one
|
||||
is *for which tasks, measured how, and at whose expense*. Posed that way, the evidence becomes
|
||||
surprisingly specific."
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
---
|
||||
name: threejs-csg
|
||||
description: >
|
||||
Build interactive 3D scenes with three.js in a single HTML file (no build step)
|
||||
and apply boolean operations (subtract, union, intersect) to meshes with
|
||||
three-bvh-csg, e.g. drilling a cylindrical hole through a cube. Use for any
|
||||
three.js, WebGL, 3D-model or CSG task.
|
||||
---
|
||||
|
||||
# three.js + CSG (boolean operations)
|
||||
|
||||
## Setup
|
||||
No npm, no bundler: ES modules are loaded from a CDN through an import map.
|
||||
[template.html](./template.html) is a complete, working page (lit scene, orbit
|
||||
controls, a cube with a cylindrical hole). Always start from it: read it with
|
||||
`load_skill("threejs-csg", "template.html")` and write it into the project
|
||||
under the requested file name, then change only `buildModel()`.
|
||||
|
||||
## Boolean operations
|
||||
Rules, pitfalls and a reference of the operations: see [csg.md](./csg.md).
|
||||
|
||||
## Import / export / text / SVG
|
||||
STL export and import, extruded SVG, solid text, and "group = union of solids
|
||||
minus holes": see [loaders.md](./loaders.md). Exporters take plain meshes with
|
||||
world matrices - not the scene, not raw Brushes.
|
||||
|
||||
## Workflow
|
||||
1. Copy the template, adapt `buildModel()` (geometry, materials, operations).
|
||||
For an editor/app around it, follow the `single-file-webapp` skill (state,
|
||||
UI wiring, test hook) and `save-load-json` for project files (meshes stay in
|
||||
a runtime map, records hold type/params/transform/hole/children).
|
||||
2. Static check: `python <single-file-webapp skill dir>/check_html.py <file>.html`.
|
||||
3. Visual/behavioural check with the `browser-test` skill (opens the file via
|
||||
`file://` - no web server needed; CDN imports work from there).
|
||||
4. Keep everything in the one HTML file unless the task asks for more.
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
# Boolean operations with three-bvh-csg
|
||||
|
||||
```js
|
||||
import { Brush, Evaluator, ADDITION, SUBTRACTION, INTERSECTION, DIFFERENCE } from 'three-bvh-csg';
|
||||
```
|
||||
(already mapped in the import map of template.html)
|
||||
|
||||
## Rules
|
||||
- CSG operands are `Brush` objects (a `Mesh` subclass): `new Brush(geometry, material)`.
|
||||
- Place the brush (`position`, `rotation`, `scale`), then call `brush.updateMatrixWorld()`;
|
||||
the evaluator works on world matrices.
|
||||
- `const ev = new Evaluator(); ev.useGroups = false;` - then the result is one mesh
|
||||
that takes the material you assign (`result.material = mat`).
|
||||
- `ev.evaluate(a, b, OP)` returns a NEW Brush. Add the result to the scene, not the operands.
|
||||
- Chain operations by reusing the result: `part = ev.evaluate(part, nextTool, SUBTRACTION)`.
|
||||
- Curved cuts need enough segments: `new THREE.CylinderGeometry(r, r, h, 48)`.
|
||||
- Make the cutting tool slightly longer than the solid (cube size 2 -> cylinder
|
||||
height 2.2). Exactly coplanar faces produce artifacts.
|
||||
- Results have no smooth normals across the cut; call `result.geometry.computeVertexNormals()`
|
||||
only if the shading looks wrong (flat shading is usually fine for CAD-like parts).
|
||||
|
||||
## Operations
|
||||
| Constant | Result |
|
||||
|---|---|
|
||||
| `ADDITION` | union of a and b |
|
||||
| `SUBTRACTION` | a minus b (holes, pockets) |
|
||||
| `INTERSECTION` | volume shared by a and b |
|
||||
| `DIFFERENCE` | symmetric difference (a xor b) |
|
||||
|
||||
## Example: cube with a through-hole
|
||||
```js
|
||||
const mat = new THREE.MeshStandardMaterial({ color: 0x4488ff, metalness: 0.2, roughness: 0.5 });
|
||||
const cube = new Brush(new THREE.BoxGeometry(2, 2, 2), mat);
|
||||
const hole = new Brush(new THREE.CylinderGeometry(0.5, 0.5, 2.2, 48), mat);
|
||||
hole.rotation.x = Math.PI / 2; // cylinder axis along z
|
||||
cube.updateMatrixWorld(); hole.updateMatrixWorld();
|
||||
|
||||
const ev = new Evaluator(); ev.useGroups = false;
|
||||
const part = ev.evaluate(cube, hole, SUBTRACTION);
|
||||
part.material = mat;
|
||||
scene.add(part);
|
||||
```
|
||||
|
|
@ -0,0 +1,82 @@
|
|||
# Loaders, exporters and text/SVG shapes
|
||||
|
||||
All of these live under `three/addons/`, which the import map of template.html
|
||||
already provides. Import from the addon file, NOT from the `THREE` namespace
|
||||
(`THREE.STLExporter` is undefined):
|
||||
```js
|
||||
import { STLExporter } from 'three/addons/exporters/STLExporter.js';
|
||||
import { STLLoader } from 'three/addons/loaders/STLLoader.js';
|
||||
import { SVGLoader } from 'three/addons/loaders/SVGLoader.js';
|
||||
import { FontLoader } from 'three/addons/loaders/FontLoader.js';
|
||||
import { TextGeometry } from 'three/addons/geometries/TextGeometry.js';
|
||||
```
|
||||
|
||||
## Export STL
|
||||
`STLExporter.parse(object3d)` traverses `object3d` and exports every `Mesh` in
|
||||
WORLD coordinates. Give it a throw-away `Group` of plain meshes - never the
|
||||
scene (grid, lights, helpers would be included) and never Brush objects with
|
||||
stale matrices:
|
||||
```js
|
||||
function toSTL(objs, binary = false) {
|
||||
const g = new THREE.Group();
|
||||
for (const o of objs) { // o.mesh: Mesh or Brush (result of CSG)
|
||||
const m = new THREE.Mesh(o.mesh.geometry); // plain Mesh, geometry is shared
|
||||
m.matrix.copy(o.mesh.matrixWorld); m.matrixAutoUpdate = false;
|
||||
g.add(m);
|
||||
}
|
||||
g.updateMatrixWorld(true);
|
||||
return new STLExporter().parse(g, { binary }); // string (ASCII) or DataView (binary)
|
||||
}
|
||||
```
|
||||
Call `obj.updateMatrixWorld(true)` after changing position/rotation and before
|
||||
exporting or evaluating CSG. Holes must already be subtracted (export the group
|
||||
results, not the hole brushes).
|
||||
|
||||
## Import STL
|
||||
```js
|
||||
pickFile('.stl', false, buf => { // ArrayBuffer (see single-file-webapp skill)
|
||||
const geo = new STLLoader().parse(buf); // handles ASCII and binary
|
||||
geo.computeVertexNormals(); geo.center(); // optional
|
||||
addObject({ type: 'stl', geometry: geo });
|
||||
});
|
||||
```
|
||||
|
||||
## Extrude an SVG
|
||||
```js
|
||||
pickFile('.svg', true, text => {
|
||||
const data = new SVGLoader().parse(text);
|
||||
const shapes = data.paths.flatMap(p => SVGLoader.createShapes(p));
|
||||
const geo = new THREE.ExtrudeGeometry(shapes, { depth: 2, bevelEnabled: false });
|
||||
geo.rotateX(Math.PI); // SVG y points down
|
||||
geo.center();
|
||||
addObject({ type: 'svg', geometry: geo });
|
||||
});
|
||||
```
|
||||
SVG units are px; scale with `geo.scale(s, s, 1)` (e.g. `s = 0.1` for mm-ish sizes).
|
||||
|
||||
## Solid text
|
||||
```js
|
||||
const FONT_URL = 'https://cdn.jsdelivr.net/npm/three@0.160.0/examples/fonts/helvetiker_regular.typeface.json';
|
||||
const font = await new FontLoader().loadAsync(FONT_URL); // load once, cache
|
||||
const geo = new TextGeometry('Hi', { font, size: 10, height: 2, curveSegments: 6 });
|
||||
geo.center();
|
||||
```
|
||||
(`height` is the extrusion depth; in three >= r163 the option is called `depth`.)
|
||||
|
||||
## Group with holes = CSG result
|
||||
```js
|
||||
function buildGroup(members) { // members: [{mesh, hole}]
|
||||
const ev = new Evaluator(); ev.useGroups = false;
|
||||
const solids = members.filter(m => !m.hole), holes = members.filter(m => m.hole);
|
||||
let result = null;
|
||||
for (const s of solids) { s.mesh.updateMatrixWorld(true);
|
||||
result = result ? ev.evaluate(result, s.mesh, ADDITION) : s.mesh; }
|
||||
for (const h of holes) { h.mesh.updateMatrixWorld(true);
|
||||
result = ev.evaluate(result, h.mesh, SUBTRACTION); }
|
||||
result.material = solids[0].mesh.material;
|
||||
return result; // a Brush; keep `members` for ungroup
|
||||
}
|
||||
```
|
||||
Operands must be `Brush` instances (create every object as a Brush from the
|
||||
start, so any object can take part in CSG later). Render holes with a
|
||||
transparent material (`transparent: true, opacity: 0.35`) until grouped.
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>three.js CSG</title>
|
||||
<style>html,body{margin:0;height:100%;overflow:hidden;background:#202428}canvas{display:block}</style>
|
||||
<script type="importmap">
|
||||
{
|
||||
"imports": {
|
||||
"three": "https://cdn.jsdelivr.net/npm/three@0.160.0/build/three.module.js",
|
||||
"three/addons/": "https://cdn.jsdelivr.net/npm/three@0.160.0/examples/jsm/",
|
||||
"three-mesh-bvh": "https://cdn.jsdelivr.net/npm/three-mesh-bvh@0.7.3/build/index.module.js",
|
||||
"three-bvh-csg": "https://cdn.jsdelivr.net/npm/three-bvh-csg@0.0.16/build/index.module.js"
|
||||
}
|
||||
}
|
||||
</script>
|
||||
</head>
|
||||
<body>
|
||||
<script type="module">
|
||||
import * as THREE from 'three';
|
||||
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';
|
||||
import { Brush, Evaluator, ADDITION, SUBTRACTION, INTERSECTION, DIFFERENCE } from 'three-bvh-csg';
|
||||
|
||||
// ---------------------------------------------------------------- scene
|
||||
const renderer = new THREE.WebGLRenderer({ antialias: true });
|
||||
renderer.setPixelRatio(window.devicePixelRatio);
|
||||
renderer.setSize(window.innerWidth, window.innerHeight);
|
||||
document.body.appendChild(renderer.domElement);
|
||||
|
||||
const scene = new THREE.Scene();
|
||||
const camera = new THREE.PerspectiveCamera(50, window.innerWidth / window.innerHeight, 0.1, 100);
|
||||
camera.position.set(4, 3, 5);
|
||||
const controls = new OrbitControls(camera, renderer.domElement);
|
||||
|
||||
scene.add(new THREE.HemisphereLight(0xffffff, 0x444444, 1.2));
|
||||
const sun = new THREE.DirectionalLight(0xffffff, 1.5);
|
||||
sun.position.set(5, 8, 4);
|
||||
scene.add(sun);
|
||||
scene.add(new THREE.GridHelper(10, 10, 0x555555, 0x333333));
|
||||
|
||||
// ---------------------------------------------------------------- model
|
||||
// Change this function only. It must return an Object3D that is added to the scene.
|
||||
function buildModel() {
|
||||
const mat = new THREE.MeshStandardMaterial({ color: 0x4488ff, metalness: 0.2, roughness: 0.5 });
|
||||
const cube = new Brush(new THREE.BoxGeometry(2, 2, 2), mat);
|
||||
const hole = new Brush(new THREE.CylinderGeometry(0.5, 0.5, 2.2, 48), mat);
|
||||
hole.rotation.x = Math.PI / 2; // cylinder axis along z
|
||||
cube.updateMatrixWorld(); hole.updateMatrixWorld();
|
||||
|
||||
const ev = new Evaluator(); ev.useGroups = false;
|
||||
const part = ev.evaluate(cube, hole, SUBTRACTION);
|
||||
part.material = mat;
|
||||
part.position.y = 1;
|
||||
return part;
|
||||
}
|
||||
scene.add(buildModel());
|
||||
|
||||
// ---------------------------------------------------------------- loop
|
||||
window.addEventListener('resize', () => {
|
||||
camera.aspect = window.innerWidth / window.innerHeight;
|
||||
camera.updateProjectionMatrix();
|
||||
renderer.setSize(window.innerWidth, window.innerHeight);
|
||||
});
|
||||
renderer.setAnimationLoop(() => { controls.update(); renderer.render(scene, camera); });
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -0,0 +1,170 @@
|
|||
---
|
||||
name: write-my-paper
|
||||
description: >
|
||||
Write a complete machine-learning conference paper in LaTeX from the experiment code and
|
||||
results in the project directory: inventory the experiments, collect the results, do a
|
||||
scientific literature search with real citations, generate figures and tables from the data,
|
||||
write all sections, compile to PDF and self-review. Use for "write my paper", "write a paper
|
||||
about these experiments", "turn my results into a conference paper", "draft the paper".
|
||||
---
|
||||
|
||||
# Write my paper
|
||||
|
||||
Goal: `paper/main.pdf`, built from `paper/main.tex` + `paper/sections/*.tex`, with figures and
|
||||
tables generated from the actual results, and citations that all come from resolved records.
|
||||
The project directory (your working directory, jcode's `./code`) holds the experiments.
|
||||
Write ONLY inside `paper/`; never edit the experiment code.
|
||||
|
||||
## Non-negotiable rules
|
||||
1. **No invented results.** Every number in the paper comes from `paper/data/summary.csv`, a
|
||||
generated table/figure, or a cited paper (cited in the same sentence). `check_paper.py`
|
||||
flags every decimal number it cannot trace - fix each one.
|
||||
2. **No invented citations.** Every `\cite` key comes from the `scientific-deep-research` skill
|
||||
(resolver + BibTeX), never from memory. Describe a cited paper only as far as you read it.
|
||||
3. **No invented experiments.** Describe only what the code does and what was run. If the code
|
||||
and the README disagree, the code wins. Missing evidence goes into Limitations, or is
|
||||
reported to the user - not papered over.
|
||||
4. **Honest framing.** Say where the method loses. "Significant" only with a test.
|
||||
|
||||
## Tools of this skill
|
||||
`load_skill` prints this skill's directory; run the scripts from there with an absolute path,
|
||||
e.g. `python <skill-dir>/results.py check paper/data/results.csv`. Use `timeout=600` for
|
||||
`run_command` calls that compile LaTeX, search literature, or run experiments. Outputs are
|
||||
truncated after 4000 characters: the scripts write full reports to files - read those with
|
||||
`read_file` (in ranges).
|
||||
- [scan_experiments.py](./scan_experiments.py) - inventory of code, configs, result files, logs,
|
||||
datasets, models, metrics -> `paper/inventory.md` / `.json`.
|
||||
- [results.py](./results.py) - tidy results CSV -> `check`, `summary`, booktabs `table`,
|
||||
`figure line|bars|scatter` (PDF + PNG preview + the plotted numbers as CSV). `melt` converts
|
||||
wide CSVs, `tb` reads TensorBoard logs. Run it without arguments for the full usage.
|
||||
- [check_paper.py](./check_paper.py) - compile + errors, undefined refs/cites, placeholders,
|
||||
number provenance, page count -> `paper/check_report.md`.
|
||||
- [template/main.tex](./template/main.tex) - generic conference skeleton (natbib, cleveref).
|
||||
- [writing-guide.md](./writing-guide.md) - what goes into every section + reviewer checklist.
|
||||
Read it with `load_skill('write-my-paper', 'writing-guide.md')` before phase 6.
|
||||
- Literature: the `scientific-deep-research` skill (`papers.py`).
|
||||
Missing packages: `pip install pandas numpy matplotlib scipy pymupdf` (and `tensorboard` only
|
||||
for TensorBoard logs). LaTeX needs `latexmk`/`pdflatex`+`bibtex` or `tectonic`; if none exists,
|
||||
say so in the final summary and still deliver the .tex sources.
|
||||
|
||||
## Progress file
|
||||
Keep `paper/STATUS.md` up to date after every phase: phase done, files produced, decisions,
|
||||
open problems. If the context was compacted or the task is resumed, read `paper/STATUS.md`
|
||||
first and continue from the first unfinished phase - don't redo finished phases.
|
||||
|
||||
## Procedure
|
||||
|
||||
### Phase 0 - Setup
|
||||
`mkdir -p paper/data paper/figures paper/tables paper/sections paper/related`, copy the
|
||||
template: `cp <skill-dir>/template/main.tex paper/`. Defaults unless the user said otherwise:
|
||||
generic conference format, double-blind ("Anonymous Authors"), ~8-9 pages main text, English.
|
||||
If the user named a venue, note its page limit and style in STATUS.md (swap the
|
||||
`\documentclass`/geometry lines for the official style file if the user provides it).
|
||||
|
||||
### Phase 1 - Understand the experiments
|
||||
1. `python <skill-dir>/scan_experiments.py . --out paper/inventory`, then read `paper/inventory.md`.
|
||||
2. Read README/docs, then every entry-point script, the model/loss code, the configs, and the
|
||||
head of each result file. Use `search_files` to locate loss functions, `argparse` defaults,
|
||||
seeds, metric computations, data splits.
|
||||
3. Write `paper/experiments.md`:
|
||||
- research question and hypothesis (one sentence each);
|
||||
- the proposed method ("ours") vs. baselines - what exactly is new, with file:line refs;
|
||||
- datasets (+ splits, preprocessing), models, training setup (optimizer, lr, batch size,
|
||||
epochs, seeds, hardware if logged), metrics + direction (higher/lower is better);
|
||||
- experiment list: main comparison, ablations, sensitivity/scaling studies, with the
|
||||
result file(s) of each and how many seeds exist;
|
||||
- gaps: experiments the code supports but whose results are missing.
|
||||
If "ours" vs. baseline cannot be determined from code/README/names, state your best
|
||||
reading and mark it as an assumption in STATUS.md and the final summary.
|
||||
4. Missing results: run an experiment only if it is clearly cheap (a few minutes). Longer runs:
|
||||
start them detached (`nohup python train.py ... > paper/data/run_x.log 2>&1 &`) only if the
|
||||
user asked for it; otherwise list the exact commands under "open problems" and continue with
|
||||
the results that exist.
|
||||
|
||||
### Phase 2 - Results into one tidy table
|
||||
1. Target: `paper/data/results.csv` with columns `method,dataset,metric,value[,seed][,step]`
|
||||
plus extra columns as needed (e.g. `variant` for ablations, `train_size`, `noise`).
|
||||
Method names exactly as they should appear in the paper (e.g. `ResNet-18 + Ours`).
|
||||
2. Wide CSVs: `results.py melt <file> --id method,dataset,seed --out paper/data/results.csv
|
||||
[--append]`. TensorBoard: `results.py tb <logdir> ...`. Anything else (JSON, logs, nested
|
||||
folders): write `paper/data/extract.py` that reads the raw files and writes the tidy CSV -
|
||||
keep it, it documents where every number came from.
|
||||
3. `results.py check paper/data/results.csv` - fix duplicates, NaNs, wrong method names.
|
||||
4. `results.py summary paper/data/results.csv [--final last|best:max|best:min]` ->
|
||||
`paper/data/summary.csv`. `--final` must match the protocol in the code (last epoch vs.
|
||||
best validation epoch). Note the choice in experiments.md.
|
||||
|
||||
### Phase 3 - Literature research
|
||||
`load_skill('scientific-deep-research')` and follow it, working inside `paper/related/`
|
||||
(`cd paper/related && python <research-skill-dir>/papers.py ...`). Search phrases must cover:
|
||||
the task/problem, the method family of "ours", each baseline, each dataset, and a **novelty
|
||||
check** (the core idea in plain words - is it already published? If yes: cite it and position
|
||||
the paper honestly as a study/extension). Deep-read 10-15 core papers; additionally resolve +
|
||||
BibTeX the original paper of every baseline, dataset, model architecture and main library
|
||||
(PyTorch, scikit-learn, ...) used - for those, the abstract is enough. Aim for 25-45 references.
|
||||
Result: `paper/related/research.md` and `paper/related/references.bib`;
|
||||
`cp paper/related/references.bib paper/references.bib`.
|
||||
|
||||
### Phase 4 - Story and outline
|
||||
Write `paper/outline.md`: working title; the one-sentence contribution; 3-4 contributions,
|
||||
each mapped to its evidence (table/figure + the numbers from summary.csv); the planned
|
||||
figures/tables (usually: 1 main results table, 1 ablation table or bar chart, 1-2 analysis
|
||||
figures such as learning curves / sensitivity / accuracy-vs-cost, optional method overview
|
||||
figure); what the related-work themes are; the limitations. The outline follows the
|
||||
evidence: if the results are mixed, the story is about when and why the method helps.
|
||||
|
||||
### Phase 5 - Figures and tables
|
||||
- Main table: `results.py table paper/data/results.csv --metrics acc:max,... --order
|
||||
<baselines...>,<ours> --highlight <ours> --scale 100 --digits 1 --caption "..." --label
|
||||
tab:main --out paper/tables/main.tex` (use `--wide` for a full-width table, `--where
|
||||
"variant=='full'"` to filter). Paste the printed caption note into `--caption`.
|
||||
- Figures: `results.py figure line ... --x step --facet dataset` (learning curves),
|
||||
`figure bars` (ablations), `figure scatter --x-metric <cost> --y-metric <quality>`
|
||||
(trade-offs), `figure line --x <hyperparameter column>` (sensitivity). Same `--highlight`
|
||||
everywhere; colours stay consistent through `paper/figures/style.json`.
|
||||
- Size figures for where they go so fonts print at 7-8 pt: `--width` = the width in the PDF in
|
||||
inches (generic template: 5.5 full width, 2.7 for half; two-column venues like ICML: 3.25
|
||||
one column, 6.75 full), then include them with `width=\linewidth`.
|
||||
- Display names: `--rename "val_acc=Accuracy,ce=Cross-entropy"` (tables and figures). Axis
|
||||
labels are plain text ("Accuracy (%)"), not LaTeX-escaped. `--where "step>=20"` / `--ylim`
|
||||
zoom in when curves overlap.
|
||||
- Look at EVERY figure: `view_image paper/figures/<name>.png`. Check: readable labels,
|
||||
legend not covering data, sensible axis ranges, units. Regenerate until it is right.
|
||||
- A method overview diagram (optional): draw it with TikZ in `paper/figures/overview.tex`
|
||||
only if it explains the method better than an equation; keep it simple.
|
||||
- Never put a number in a table by hand. Re-run the command instead.
|
||||
|
||||
### Phase 6 - Write the sections
|
||||
Read `writing-guide.md` first. Write one file per section into `paper/sections/`, in this order
|
||||
(evidence first, framing last): `method`, `experimental_setup`, `results`, `related_work`,
|
||||
`introduction`, `discussion`, `conclusion`, `statements`, `appendix`, `abstract`; then set the
|
||||
title and `\ours` macro in `main.tex`. While writing `method` and `experimental_setup`, keep the
|
||||
code open - equations and hyperparameters must match it. While writing `results`, keep
|
||||
`paper/data/summary.csv` and the printed table numbers open and copy numbers from there.
|
||||
Insert generated tables with `\input{tables/main}` and figures with
|
||||
`\includegraphics[width=\linewidth]{figures/<name>.pdf}`.
|
||||
|
||||
### Phase 7 - Compile and check
|
||||
`python <skill-dir>/check_paper.py paper/main.tex [--max-pages 9]` (timeout=600). Fix every
|
||||
ERROR, then the warnings: untraced numbers (trace or remove), unreferenced figures, overfull
|
||||
boxes, uncited bib entries (cite or remove). Repeat until 0 errors. Then look at the PDF
|
||||
with `view_pdf_page paper/main.pdf <page>` for every page: float placement, figure size,
|
||||
table width, orphaned headings. The page limit counts main text only (references and appendix
|
||||
usually excluded) - cut text, not evidence, if over.
|
||||
|
||||
### Phase 8 - Self-review and hand-over
|
||||
Review the paper as a critical reviewer with the checklist in writing-guide.md; write
|
||||
`paper/review.md` (strengths, weaknesses, questions a reviewer will ask). Fix what can be
|
||||
fixed by writing; list what needs new experiments. Final run of check_paper.py. Final answer
|
||||
to the user (short): where the PDF is, page count, number of references, the main claims with
|
||||
their numbers, assumptions you made, and the open problems / suggested extra experiments.
|
||||
|
||||
## Final layout
|
||||
```
|
||||
paper/
|
||||
main.tex main.pdf references.bib STATUS.md check_report.md review.md
|
||||
inventory.md experiments.md outline.md
|
||||
sections/*.tex tables/*.tex figures/*.pdf|png|data.csv figures/style.json
|
||||
data/results.csv data/summary.csv [data/extract.py]
|
||||
related/research.md related/references.bib related/papers/ related/candidates.json
|
||||
```
|
||||
|
|
@ -0,0 +1,290 @@
|
|||
#!/usr/bin/env python3
|
||||
"""check_paper.py - compile the LaTeX paper and check it the way a strict co-author would.
|
||||
|
||||
python check_paper.py paper/main.tex [--summary paper/data/summary.csv] [--max-pages 9] [--no-compile]
|
||||
|
||||
1. Compiles (latexmk -> pdflatex+bibtex -> tectonic, whichever exists) and extracts LaTeX errors.
|
||||
2. Log: undefined citations/references, duplicate labels, missing files, overfull boxes > 10pt.
|
||||
3. Source (main.tex + every \\input/\\include): \\cite keys missing from the .bib, uncited .bib entries,
|
||||
\\ref targets without \\label, figures/tables never referenced, leftover TODO/XXX/??/placeholder text,
|
||||
abstract length, bib entries without author/title/year.
|
||||
4. Number provenance: every decimal number / percentage in the prose (tables generated by results.py
|
||||
excluded) is looked up in summary.csv (mean, std, ci95, min, max; scaled x1 and x100, rounded to the
|
||||
shown digits). Numbers not found are listed - each one must be traced to data, a cited paper, or removed.
|
||||
5. Page count (pdfinfo or pymupdf), page on which the references start.
|
||||
Prints a compact report and writes the full one to <tex dir>/check_report.md. Exit code 1 if errors.
|
||||
"""
|
||||
import csv, re, shutil, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
CITE_RE = re.compile(r"\\(?:no)?cite[a-zA-Z]*\*?(?:\[[^\]]*\]){0,2}\{([^}]*)\}")
|
||||
REF_RE = re.compile(r"\\(?:ref|cref|Cref|autoref|eqref|pageref|nameref|vref)\*?\{([^}]*)\}")
|
||||
LABEL_RE = re.compile(r"\\label\{([^}]*)\}")
|
||||
INPUT_RE = re.compile(r"\\(?:input|include|subfile)\{([^}]*)\}")
|
||||
BIB_RE = re.compile(r"\\(?:bibliography|addbibresource)\{([^}]*)\}")
|
||||
TODO_RE = re.compile(r"\bTODO\b|\bXXX\b|\bFIXME\b|\?\?\?|\[citation needed\]|lorem ipsum|\bTBD\b|<[A-Z][A-Z _-]{3,}>", re.I)
|
||||
NUM_RE = re.compile(r"(?<![\w.\\{-])(\d+\.\d+|\d+(?=\s*\\?%))(\s*\\?%)?")
|
||||
|
||||
|
||||
def strip_comments(t):
|
||||
return re.sub(r"(?<!\\)%.*", "", t)
|
||||
|
||||
|
||||
def gather(tex, seen=None):
|
||||
"""Concatenated source of tex and its inputs: list of (path, text)."""
|
||||
seen = seen or set()
|
||||
p = tex if tex.suffix else tex.with_suffix(".tex")
|
||||
if p in seen or not p.is_file():
|
||||
return []
|
||||
seen.add(p)
|
||||
text = strip_comments(p.read_text(encoding="utf-8", errors="replace"))
|
||||
out = [(p, text)]
|
||||
for inc in INPUT_RE.findall(text):
|
||||
q = (TEX_DIR / inc)
|
||||
out += gather(q if q.suffix else q.with_suffix(".tex"), seen)
|
||||
return out
|
||||
|
||||
|
||||
def expand(p, depth=0):
|
||||
"""Source of p with every \\input/\\include replaced by the included text (for abstract etc.)."""
|
||||
p = p if p.suffix else p.with_suffix(".tex")
|
||||
if depth > 10 or not p.is_file():
|
||||
return ""
|
||||
text = strip_comments(p.read_text(encoding="utf-8", errors="replace"))
|
||||
return INPUT_RE.sub(lambda m: expand(TEX_DIR / m.group(1), depth + 1), text)
|
||||
|
||||
|
||||
def run(cmd, cwd, timeout=300):
|
||||
try:
|
||||
r = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=timeout)
|
||||
return r.returncode, r.stdout + r.stderr
|
||||
except subprocess.TimeoutExpired:
|
||||
return 1, f"TIMEOUT: {' '.join(cmd)}"
|
||||
|
||||
|
||||
def compile_tex(tex):
|
||||
d, name = tex.parent, tex.stem
|
||||
if shutil.which("latexmk"):
|
||||
code, out = run(["latexmk", "-pdf", "-interaction=nonstopmode", "-halt-on-error", "-file-line-error", tex.name], d)
|
||||
how = "latexmk"
|
||||
elif shutil.which("pdflatex"):
|
||||
how, out, code = "pdflatex+bibtex", "", 0
|
||||
for cmd in (["pdflatex"], ["bibtex"], ["pdflatex"], ["pdflatex"]):
|
||||
arg = name if cmd[0] == "bibtex" else tex.name
|
||||
extra = [] if cmd[0] == "bibtex" else ["-interaction=nonstopmode", "-halt-on-error", "-file-line-error"]
|
||||
c, o = run(cmd + extra + [arg], d)
|
||||
out += o
|
||||
if c and cmd[0] != "bibtex":
|
||||
code = c; break
|
||||
elif shutil.which("tectonic"):
|
||||
code, out = run(["tectonic", "--keep-logs", tex.name], d)
|
||||
how = "tectonic"
|
||||
else:
|
||||
return None, "no LaTeX found: install TeX Live (latexmk, pdflatex, bibtex) or tectonic", []
|
||||
log = d / f"{name}.log"
|
||||
logtext = log.read_text(encoding="utf-8", errors="replace") if log.is_file() else out
|
||||
errors = []
|
||||
lines = logtext.splitlines()
|
||||
for i, l in enumerate(lines):
|
||||
if l.startswith("!") or re.match(r"^[^:\s]+\.tex:\d+: ", l):
|
||||
errors.append(" | ".join(x.strip() for x in lines[i:i + 3] if x.strip())[:300])
|
||||
if code and not errors:
|
||||
errors.append(out.strip().splitlines()[-1][:300] if out.strip() else f"{how} failed")
|
||||
return how, logtext, errors[:15]
|
||||
|
||||
|
||||
def log_warnings(logtext):
|
||||
w = []
|
||||
w += [f"undefined citation: {k}" for k in sorted(set(re.findall(r"Citation [`']([^']+)' (?:on page \d+ )?undefined", logtext)))]
|
||||
w += [f"undefined reference: {k}" for k in sorted(set(re.findall(r"Reference [`']([^']+)' on page \d+ undefined", logtext)))]
|
||||
w += [f"label multiply defined: {k}" for k in sorted(set(re.findall(r"Label [`']([^']+)' multiply defined", logtext)))]
|
||||
w += [f"missing file: {k}" for k in sorted(set(re.findall(r"File [`']([^']+)' not found", logtext)))]
|
||||
for m in re.finditer(r"Overfull \\hbox \((\d+\.?\d*)pt too wide\)[^\n]*lines? (\d+)", logtext):
|
||||
if float(m.group(1)) > 10:
|
||||
w.append(f"overfull hbox {m.group(1)}pt near line {m.group(2)}")
|
||||
if "Rerun to get" in logtext[-3000:]:
|
||||
w.append("LaTeX asks for a rerun (cross-references may be stale)")
|
||||
return w
|
||||
|
||||
|
||||
def bib_entries(paths):
|
||||
entries = {}
|
||||
for b in paths:
|
||||
if not b.is_file():
|
||||
continue
|
||||
text = b.read_text(encoding="utf-8", errors="replace")
|
||||
for m in re.finditer(r"@(\w+)\s*\{\s*([^,\s]+)\s*,(.*?)(?=\n@|\Z)", text, re.S):
|
||||
if m.group(1).lower() in ("string", "comment", "preamble"):
|
||||
continue
|
||||
body = m.group(3).lower()
|
||||
entries[m.group(2)] = {"type": m.group(1).lower(),
|
||||
"missing": [f for f in ("author", "title", "year") if not re.search(rf"\b{f}\s*=", body)]}
|
||||
return entries
|
||||
|
||||
|
||||
def summary_numbers(path):
|
||||
vals = []
|
||||
if not path or not Path(path).is_file():
|
||||
return None
|
||||
with open(path, newline="") as f:
|
||||
for row in csv.DictReader(f):
|
||||
for k in ("mean", "std", "ci95", "min", "max"):
|
||||
try:
|
||||
v = float(row.get(k, ""))
|
||||
except ValueError:
|
||||
continue
|
||||
vals += [v, v * 100]
|
||||
# also plotted numbers next to figures
|
||||
return vals
|
||||
|
||||
|
||||
def number_found(txt, vals, extra):
|
||||
if txt in extra:
|
||||
return True
|
||||
digits = len(txt.split(".")[1]) if "." in txt else 0
|
||||
x = float(txt)
|
||||
tol = 0.5 * 10 ** (-digits) + 1e-9
|
||||
return any(abs(v - x) <= tol for v in vals)
|
||||
|
||||
|
||||
def prose_numbers(files):
|
||||
out = []
|
||||
for p, text in files:
|
||||
if "generated by results.py" in text:
|
||||
continue
|
||||
body = re.sub(r"\\begin\{(table\*?|tabular|equation\*?|align\*?)\}.*?\\end\{\1\}", " ", text, flags=re.S)
|
||||
body = re.sub(r"\$[^$]*\$", lambda m: m.group(0) if re.search(r"\d\.\d", m.group(0)) else " ", body)
|
||||
body = re.sub(r"\\(?:includegraphics|vspace|hspace|setlength|resizebox|scalebox)\*?(\[[^\]]*\])?\{[^}]*\}", " ", body)
|
||||
body = re.sub(r"\[[^\]]*(?:width|height|scale)=[^\]]*\]", " ", body)
|
||||
body = re.sub(r"\\documentclass\[[^\]]*\]|\\usepackage\[[^\]]*\]", " ", body)
|
||||
for i, line in enumerate(body.splitlines(), 1):
|
||||
for m in NUM_RE.finditer(line):
|
||||
num = m.group(1)
|
||||
if not m.group(2) and "." not in num:
|
||||
continue
|
||||
if re.fullmatch(r"(19|20)\d\d", num):
|
||||
continue
|
||||
out.append((p.name, i, num + ("%" if m.group(2) else ""), line.strip()[:110]))
|
||||
return out
|
||||
|
||||
|
||||
def pdf_info(pdf):
|
||||
pages, ref_page = None, None
|
||||
if shutil.which("pdfinfo"):
|
||||
c, o = run(["pdfinfo", str(pdf)], pdf.parent, 30)
|
||||
m = re.search(r"Pages:\s+(\d+)", o)
|
||||
pages = int(m.group(1)) if m else None
|
||||
if shutil.which("pdftotext") and pages:
|
||||
for pg in range(1, pages + 1):
|
||||
c, o = run(["pdftotext", "-f", str(pg), "-l", str(pg), str(pdf), "-"], pdf.parent, 30)
|
||||
if re.search(r"(?m)^\s*(References|Bibliography)\s*$", o):
|
||||
ref_page = pg; break
|
||||
if pages is None:
|
||||
try:
|
||||
import pymupdf as fitz
|
||||
except ImportError:
|
||||
try:
|
||||
import fitz
|
||||
except ImportError:
|
||||
return None, None
|
||||
with fitz.open(pdf) as doc:
|
||||
pages = doc.page_count
|
||||
for i, page in enumerate(doc):
|
||||
if re.search(r"(?m)^\s*(References|Bibliography)\s*$", page.get_text()):
|
||||
ref_page = i + 1; break
|
||||
return pages, ref_page
|
||||
|
||||
|
||||
def main(argv):
|
||||
global TEX_DIR
|
||||
if not argv or argv[0] in ("-h", "--help"):
|
||||
print(__doc__); return 0
|
||||
tex = Path(argv[0]).resolve()
|
||||
TEX_DIR = tex.parent
|
||||
opt = lambda k, d=None: argv[argv.index(k) + 1] if k in argv else d
|
||||
summary = opt("--summary", str(TEX_DIR / "data" / "summary.csv"))
|
||||
max_pages = int(opt("--max-pages", 0) or 0)
|
||||
report, errors, warns = [], [], []
|
||||
|
||||
if "--no-compile" not in argv:
|
||||
how, logtext, errs = compile_tex(tex)
|
||||
if how is None:
|
||||
warns.append(logtext)
|
||||
else:
|
||||
report.append(f"compiled with {how}: {'FAILED' if errs else 'ok'}")
|
||||
errors += [f"LaTeX: {e}" for e in errs]
|
||||
warns += log_warnings(logtext)
|
||||
|
||||
files = gather(tex)
|
||||
src = "\n".join(t for _, t in files)
|
||||
cites = {k.strip() for g in CITE_RE.findall(src) for k in g.split(",") if k.strip()}
|
||||
bibs = [TEX_DIR / (b.strip() if b.strip().endswith(".bib") else b.strip() + ".bib")
|
||||
for g in BIB_RE.findall(src) for b in g.split(",")]
|
||||
entries = bib_entries(bibs)
|
||||
if not bibs:
|
||||
warns.append("no \\bibliography{...} found")
|
||||
for k in sorted(cites - set(entries)):
|
||||
errors.append(f"cited but not in .bib: {k}")
|
||||
uncited = sorted(set(entries) - cites)
|
||||
if uncited:
|
||||
warns.append(f"{len(uncited)} .bib entries never cited (fine with natbib, but check): {', '.join(uncited[:15])}")
|
||||
for k, e in entries.items():
|
||||
if e["missing"]:
|
||||
warns.append(f"bib entry {k} lacks {', '.join(e['missing'])}")
|
||||
labels = LABEL_RE.findall(src)
|
||||
refs = {k.strip() for g in REF_RE.findall(src) for k in g.split(",")}
|
||||
for k in sorted(refs - set(labels)):
|
||||
errors.append(f"\\ref to undefined label: {k}")
|
||||
for k in sorted(set(l for l in labels if l.startswith(("fig:", "tab:"))) - refs):
|
||||
warns.append(f"{k} is never referenced in the text")
|
||||
for p, text in files:
|
||||
for i, line in enumerate(text.splitlines(), 1):
|
||||
if TODO_RE.search(line):
|
||||
errors.append(f"placeholder in {p.name}:{i}: {line.strip()[:100]}")
|
||||
m = re.search(r"\\begin\{abstract\}(.*?)\\end\{abstract\}", expand(tex), re.S)
|
||||
if m:
|
||||
words = len(re.sub(r"\\[a-zA-Z]+|[{}$]", " ", m.group(1)).split())
|
||||
report.append(f"abstract: {words} words")
|
||||
if words > 250:
|
||||
warns.append(f"abstract has {words} words (> 250)")
|
||||
else:
|
||||
warns.append("no abstract environment")
|
||||
|
||||
vals = summary_numbers(summary)
|
||||
nums = prose_numbers(files)
|
||||
if vals is None:
|
||||
warns.append(f"no summary file {summary} - number provenance not checked (run results.py summary)")
|
||||
else:
|
||||
plotted = set()
|
||||
for dcsv in (TEX_DIR / "figures").glob("*.data.csv"):
|
||||
plotted |= set(re.findall(r"-?\d+\.\d+", dcsv.read_text()))
|
||||
unknown = [n for n in nums if not number_found(n[2].rstrip("%"), vals, plotted)]
|
||||
report.append(f"number provenance: {len(nums) - len(unknown)}/{len(nums)} decimal numbers in the prose match summary.csv")
|
||||
for f, i, num, line in unknown[:25]:
|
||||
warns.append(f"number {num} ({f}:{i}) not in summary.csv - from a cited paper (cite it) or a setting? else fix: {line}")
|
||||
|
||||
pdf = tex.with_suffix(".pdf")
|
||||
if pdf.is_file():
|
||||
pages, refp = pdf_info(pdf)
|
||||
if pages:
|
||||
body = refp if refp else pages # references start on that page: count it as text
|
||||
report.append(f"pdf: {pages} pages; references start on page {refp or '?'} (main text ~{body} pages)")
|
||||
if max_pages and body > max_pages:
|
||||
errors.append(f"main text ~{body} pages > limit {max_pages}")
|
||||
elif "--no-compile" not in argv:
|
||||
errors.append("no PDF produced")
|
||||
|
||||
full = ["# check_paper report", ""] + report + ["", f"## Errors ({len(errors)})"] + [f"- {e}" for e in errors] + \
|
||||
["", f"## Warnings ({len(warns)})"] + [f"- {w}" for w in warns]
|
||||
(TEX_DIR / "check_report.md").write_text("\n".join(full) + "\n", encoding="utf-8")
|
||||
print("\n".join(report))
|
||||
print(f"\nERRORS ({len(errors)}):" + ("".join(f"\n - {e}" for e in errors[:20]) or " none"))
|
||||
print(f"WARNINGS ({len(warns)}):" + ("".join(f"\n - {w[:220]}" for w in warns[:20]) or " none"))
|
||||
if len(errors) > 20 or len(warns) > 20:
|
||||
print(f"... full list in {TEX_DIR / 'check_report.md'}")
|
||||
return 1 if errors else 0
|
||||
|
||||
|
||||
TEX_DIR = Path(".")
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
|
|
@ -0,0 +1,536 @@
|
|||
#!/usr/bin/env python3
|
||||
"""results.py - turn experiment results into paper-ready LaTeX tables and PDF figures.
|
||||
|
||||
All commands work on ONE tidy CSV (long format), one row per measured value:
|
||||
method,dataset,metric,value[,seed][,step][,<any extra column, e.g. noise, train_size, variant>]
|
||||
|
||||
python results.py melt wide.csv --id method,dataset,seed[,step] [--metrics acc,f1] --out paper/data/results.csv [--append]
|
||||
python results.py tb LOGDIR --method-level 1 [--dataset NAME] --out paper/data/results.csv [--append]
|
||||
python results.py check paper/data/results.csv
|
||||
python results.py summary paper/data/results.csv [--final last|best:max|best:min] [--out paper/data/summary.csv]
|
||||
python results.py table paper/data/results.csv --metrics acc:max[,loss:min] --out paper/tables/main.tex
|
||||
[--rows method] [--cols dataset] [--order ours,base1,...] [--highlight ours] [--scale 100] [--digits 1]
|
||||
[--where "variant==full"] [--final last] [--caption "..."] [--label tab:main] [--no-std] [--wide]
|
||||
python results.py figure line paper/data/results.csv --metric val_acc --x step [--facet dataset] --out paper/figures/curves.pdf
|
||||
python results.py figure bars paper/data/results.csv --metric acc [--group dataset] --out paper/figures/bars.pdf
|
||||
python results.py figure scatter paper/data/results.csv --x-metric time --y-metric acc [--facet dataset] --out paper/figures/tradeoff.pdf
|
||||
common figure options: [--order a,b,c] [--highlight ours] [--width single|double|<inches>] [--height 2.0]
|
||||
[--where "dataset=='CIFAR-10'"] [--ylabel "..."] [--xlabel "..."] [--logx] [--logy] [--ylim 60,80] [--xlim 0,50] [--scale 100] [--final last]
|
||||
|
||||
melt wide table (one column per metric) -> tidy rows. tb TensorBoard scalars -> tidy rows (pip install tensorboard).
|
||||
check validates the tidy file and prints the coverage: methods x datasets x metrics, seeds per cell, gaps.
|
||||
summary final value per run (last step, or best step), then mean/std/n/95%-CI per method x dataset x metric.
|
||||
Writes summary.csv - the ONLY allowed source for numbers quoted in the paper text.
|
||||
table booktabs table: mean +- std over seeds, best per column in bold, second underlined, cells that are
|
||||
not significantly worse than the best (Welch t-test, p >= 0.05, needs >= 3 seeds) marked with a dagger.
|
||||
figure vector PDF (+ PNG preview for view_image) in a consistent, colour-blind-safe style; each method keeps
|
||||
the same colour and marker in every figure (paper/figures/style.json). Shaded band = +-1 std over seeds.
|
||||
--rename "val_acc=Accuracy,ce=Cross-entropy" sets display names (methods, datasets, metrics, columns) in
|
||||
tables and figures without touching the data. Axis labels are plain text: write "Accuracy (%)", not "\\%".
|
||||
Every table/figure command also prints the numbers it used, so the text can quote them exactly.
|
||||
Requires pandas, numpy, matplotlib (scipy optional for the t-test).
|
||||
"""
|
||||
import json, math, re, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
except ImportError:
|
||||
sys.exit("pip install pandas numpy matplotlib scipy")
|
||||
|
||||
REQUIRED = ["method", "dataset", "metric", "value"]
|
||||
# Okabe-Ito without black/yellow (validated: CVD-separable; markers give the secondary encoding).
|
||||
PALETTE = ["#0072B2", "#E69F00", "#009E73", "#D55E00", "#CC79A7", "#56B4E9"]
|
||||
MARKERS = ["o", "s", "^", "D", "v", "P", "X", "*"]
|
||||
LINESTYLES = ["-", "--", ":", "-."]
|
||||
STYLE_FILE = Path("paper/figures/style.json")
|
||||
RENAME = {} # display names, from --rename "val_acc=Accuracy,ce=Cross-entropy"
|
||||
|
||||
|
||||
def disp(x):
|
||||
return RENAME.get(str(x), str(x))
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- arguments
|
||||
def parse(argv):
|
||||
pos, opt, i = [], {}, 0
|
||||
while i < len(argv):
|
||||
a = argv[i]
|
||||
if a.startswith("--"):
|
||||
if i + 1 < len(argv) and not argv[i + 1].startswith("--"):
|
||||
opt[a[2:]] = argv[i + 1]; i += 2
|
||||
else:
|
||||
opt[a[2:]] = True; i += 1
|
||||
else:
|
||||
pos.append(a); i += 1
|
||||
return pos, opt
|
||||
|
||||
|
||||
def csv_list(s):
|
||||
return [x.strip() for x in str(s).split(",") if x.strip()] if s and s is not True else []
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- loading
|
||||
def load(path, where=None):
|
||||
df = pd.read_csv(path)
|
||||
miss = [c for c in REQUIRED if c not in df.columns]
|
||||
if miss:
|
||||
sys.exit(f"{path}: missing columns {miss}. Required: {REQUIRED} (+ optional seed, step, extras)")
|
||||
df["value"] = pd.to_numeric(df["value"], errors="coerce")
|
||||
for c in ("method", "dataset", "metric"):
|
||||
df[c] = df[c].astype(str)
|
||||
if "seed" not in df.columns:
|
||||
df["seed"] = 0
|
||||
if where:
|
||||
df = df.query(where, engine="python")
|
||||
if df.empty:
|
||||
sys.exit(f"--where {where!r} selects no rows")
|
||||
return df
|
||||
|
||||
|
||||
def extras(df):
|
||||
return [c for c in df.columns if c not in REQUIRED + ["seed", "step"]]
|
||||
|
||||
|
||||
def final_values(df, final="last"):
|
||||
"""One value per run (= method, dataset, metric, seed, extras): last step, or best step."""
|
||||
keys = ["method", "dataset", "metric", "seed"] + extras(df)
|
||||
df = df.dropna(subset=["value"])
|
||||
if "step" not in df.columns or df["step"].isna().all():
|
||||
return df.groupby(keys, dropna=False, as_index=False)["value"].mean()
|
||||
df = df.sort_values("step")
|
||||
if final == "last":
|
||||
return df.groupby(keys, dropna=False, as_index=False)["value"].last()
|
||||
mode = final.split(":")[-1]
|
||||
agg = "max" if mode == "max" else "min"
|
||||
return df.groupby(keys, dropna=False, as_index=False)["value"].agg(agg)
|
||||
|
||||
|
||||
def aggregate(fin, by):
|
||||
g = fin.groupby(by, dropna=False)["value"]
|
||||
s = g.agg(["mean", "std", "count", "min", "max"]).reset_index().rename(columns={"count": "n"})
|
||||
s["std"] = s["std"].fillna(0.0)
|
||||
tcrit = s["n"].map(lambda n: _t975(n - 1) if n > 1 else float("nan"))
|
||||
s["ci95"] = tcrit * s["std"] / np.sqrt(s["n"])
|
||||
return s
|
||||
|
||||
|
||||
def _t975(dof):
|
||||
try:
|
||||
from scipy import stats
|
||||
return float(stats.t.ppf(0.975, dof))
|
||||
except ImportError:
|
||||
table = {1: 12.71, 2: 4.30, 3: 3.18, 4: 2.78, 5: 2.57, 6: 2.45, 7: 2.36, 8: 2.31, 9: 2.26, 10: 2.23}
|
||||
return table.get(dof, 1.96)
|
||||
|
||||
|
||||
def welch_p(a, b):
|
||||
if len(a) < 3 or len(b) < 3:
|
||||
return None
|
||||
try:
|
||||
from scipy import stats
|
||||
return float(stats.ttest_ind(a, b, equal_var=False).pvalue)
|
||||
except ImportError:
|
||||
return None
|
||||
|
||||
|
||||
def ordered(values, order, highlight=None):
|
||||
vals = list(dict.fromkeys(values))
|
||||
first = [v for v in order if v in vals]
|
||||
rest = sorted(v for v in vals if v not in first)
|
||||
out = first + rest
|
||||
if highlight and highlight in out and not order:
|
||||
out.remove(highlight); out.insert(0, highlight)
|
||||
return out
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- melt / tb
|
||||
def cmd_melt(pos, opt):
|
||||
df = pd.read_csv(pos[0])
|
||||
ids = csv_list(opt.get("id"))
|
||||
miss = [c for c in ids if c not in df.columns]
|
||||
if miss:
|
||||
sys.exit(f"--id columns not in file: {miss}; columns are {list(df.columns)}")
|
||||
metrics = csv_list(opt.get("metrics")) or [c for c in df.columns if c not in ids and pd.api.types.is_numeric_dtype(df[c])]
|
||||
long = df.melt(id_vars=ids, value_vars=metrics, var_name="metric", value_name="value")
|
||||
for c in ("method", "dataset"):
|
||||
if c not in long.columns:
|
||||
if opt.get(c):
|
||||
long[c] = opt[c]
|
||||
else:
|
||||
sys.exit(f"no '{c}' column - add it to --id, or give a constant with --{c} NAME")
|
||||
write_tidy(long, opt)
|
||||
|
||||
|
||||
def cmd_tb(pos, opt):
|
||||
try:
|
||||
from tensorboard.backend.event_processing.event_accumulator import EventAccumulator
|
||||
except ImportError:
|
||||
sys.exit("pip install tensorboard")
|
||||
root, level = Path(pos[0]), int(opt.get("method-level", 1))
|
||||
rows = []
|
||||
for ev in sorted(root.rglob("*tfevents*")):
|
||||
run = ev.parent.relative_to(root).parts
|
||||
method = run[min(level, len(run)) - 1] if run else root.name
|
||||
seed_m = re.search(r"seed[_-]?(\d+)", str(ev.parent))
|
||||
acc = EventAccumulator(str(ev.parent)); acc.Reload()
|
||||
for tag in acc.Tags().get("scalars", []):
|
||||
for e in acc.Scalars(tag):
|
||||
rows.append({"method": method, "dataset": opt.get("dataset", "default"), "metric": tag,
|
||||
"value": e.value, "seed": int(seed_m.group(1)) if seed_m else 0, "step": e.step,
|
||||
"run": "/".join(run)})
|
||||
if not rows:
|
||||
sys.exit("no scalars found")
|
||||
write_tidy(pd.DataFrame(rows).drop_duplicates(), opt)
|
||||
|
||||
|
||||
def write_tidy(df, opt):
|
||||
out = Path(opt.get("out", "paper/data/results.csv"))
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
if opt.get("append") and out.exists():
|
||||
df = pd.concat([pd.read_csv(out), df], ignore_index=True)
|
||||
cols = REQUIRED + [c for c in ("seed", "step") if c in df.columns] + [c for c in df.columns if c not in REQUIRED + ["seed", "step"]]
|
||||
df[cols].to_csv(out, index=False)
|
||||
print(f"wrote {out}: {len(df)} rows, metrics {sorted(df['metric'].astype(str).unique())[:15]}")
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- check / summary
|
||||
def cmd_check(pos, opt):
|
||||
df = load(pos[0], opt.get("where"))
|
||||
ex = extras(df)
|
||||
print(f"{pos[0]}: {len(df)} rows; extra columns: {ex or '-'}; steps: {'yes' if 'step' in df.columns else 'no'}")
|
||||
nan = df["value"].isna().sum()
|
||||
if nan:
|
||||
print(f"WARNING: {nan} non-numeric/NaN values")
|
||||
keys = ["method", "dataset", "metric", "seed"] + ex + (["step"] if "step" in df.columns else [])
|
||||
dup = df.duplicated(keys).sum()
|
||||
if dup:
|
||||
print(f"WARNING: {dup} duplicate rows for the same run/step - add a distinguishing column (e.g. variant) or deduplicate")
|
||||
for metric, d in df.groupby("metric"):
|
||||
seeds = d.groupby(["method", "dataset"])["seed"].nunique().unstack("dataset")
|
||||
print(f"\nmetric '{metric}': seeds per method x dataset (blank = missing)")
|
||||
print(seeds.fillna(0).astype(int).replace(0, "").to_string())
|
||||
n = df.groupby(["method", "dataset", "metric"])["seed"].nunique()
|
||||
if (n < 3).any():
|
||||
print("\nNOTE: some cells have < 3 seeds - no significance tests there; say so in the paper.")
|
||||
methods, datasets = df["method"].unique(), df["dataset"].unique()
|
||||
print(f"\n{len(methods)} methods: {', '.join(map(str, methods))}\n{len(datasets)} datasets: {', '.join(map(str, datasets))}")
|
||||
|
||||
|
||||
def cmd_summary(pos, opt):
|
||||
df = load(pos[0], opt.get("where"))
|
||||
fin = final_values(df, opt.get("final", "last"))
|
||||
s = aggregate(fin, ["method", "dataset", "metric"] + extras(df))
|
||||
out = Path(opt.get("out", Path(pos[0]).with_name("summary.csv")))
|
||||
s.to_csv(out, index=False, float_format="%.6g")
|
||||
with pd.option_context("display.max_rows", 200, "display.width", 200):
|
||||
print(s.to_string(index=False, float_format=lambda v: f"{v:.4g}"))
|
||||
print(f"\nwrote {out} ({len(s)} cells) - quote numbers in the text only from this file or the generated tables")
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- table
|
||||
def fmt(v, digits):
|
||||
return "--" if v is None or (isinstance(v, float) and math.isnan(v)) else f"{v:.{digits}f}"
|
||||
|
||||
|
||||
def tex_escape(s):
|
||||
return re.sub(r"([&%$#_{}])", r"\\\1", str(s))
|
||||
|
||||
|
||||
def arrow(d):
|
||||
return r"$\uparrow$" if d == "max" else r"$\downarrow$"
|
||||
|
||||
|
||||
def cmd_table(pos, opt):
|
||||
df = load(pos[0], opt.get("where"))
|
||||
specs = [m.split(":") for m in csv_list(opt.get("metrics"))]
|
||||
if not specs:
|
||||
sys.exit("--metrics acc:max,loss:min is required (max = higher is better)")
|
||||
for sp in specs:
|
||||
if len(sp) != 2 or sp[1] not in ("max", "min"):
|
||||
sys.exit(f"bad metric spec {':'.join(sp)} - use name:max or name:min")
|
||||
if sp[0] not in set(df["metric"]):
|
||||
sys.exit(f"metric '{sp[0]}' not in data; available: {sorted(df['metric'].unique())}")
|
||||
rows_by, cols_by = opt.get("rows", "method"), opt.get("cols", "dataset")
|
||||
scale, digits = float(opt.get("scale", 1)), int(opt.get("digits", 2))
|
||||
show_std = not opt.get("no-std")
|
||||
fin = final_values(df[df["metric"].isin([s[0] for s in specs])], opt.get("final", "last"))
|
||||
fin["value"] *= scale
|
||||
row_vals = ordered(fin[rows_by], csv_list(opt.get("order")), opt.get("highlight"))
|
||||
col_vals = ordered(fin[cols_by], csv_list(opt.get("col-order")))
|
||||
multi_metric = len(specs) > 1
|
||||
columns = [(c, m, d) for c in col_vals for m, d in specs]
|
||||
cell, raw = {}, {}
|
||||
for (r, c, m), g in fin.groupby([rows_by, cols_by, "metric"]):
|
||||
cell[(r, c, m)] = (g["value"].mean(), g["value"].std(ddof=1) if len(g) > 1 else 0.0, len(g))
|
||||
raw[(r, c, m)] = g["value"].to_numpy()
|
||||
marks = {}
|
||||
for c, m, d in columns:
|
||||
present = [(r, cell[(r, c, m)][0]) for r in row_vals if (r, c, m) in cell]
|
||||
if not present:
|
||||
continue
|
||||
ranked = sorted(present, key=lambda x: -x[1] if d == "max" else x[1])
|
||||
best = ranked[0][0]
|
||||
marks[(best, c, m)] = "best"
|
||||
if len(ranked) > 1 and ranked[1][1] != ranked[0][1]:
|
||||
marks[(ranked[1][0], c, m)] = "second"
|
||||
for r, _ in ranked[1:]:
|
||||
p = welch_p(raw[(r, c, m)], raw[(best, c, m)])
|
||||
if p is not None and p >= 0.05:
|
||||
marks[(r, c, m, "tie")] = True
|
||||
ncol = len(columns)
|
||||
L = [r"% generated by results.py - do not edit by hand; regenerate instead", r"\begin{table*}[t]" if opt.get("wide") else r"\begin{table}[t]",
|
||||
r"\centering", r"\caption{" + opt.get("caption", "TODO caption.") + "}", r"\label{" + opt.get("label", "tab:main") + "}",
|
||||
r"\resizebox{\linewidth}{!}{%" if ncol > 4 else "", r"\begin{tabular}{l" + "c" * ncol + "}", r"\toprule"]
|
||||
if multi_metric and len(col_vals) > 1:
|
||||
L.append(" & " + " & ".join(rf"\multicolumn{{{len(specs)}}}{{c}}{{{tex_escape(disp(c))}}}" for c in col_vals) + r" \\")
|
||||
L.append("".join(rf"\cmidrule(lr){{{2 + i * len(specs)}-{1 + (i + 1) * len(specs)}}}" for i in range(len(col_vals))))
|
||||
L.append(tex_escape(rows_by.capitalize()) + " & " + " & ".join(
|
||||
tex_escape(disp(m)) + " " + arrow(d) for _, m, d in columns) + r" \\")
|
||||
elif multi_metric:
|
||||
L.append(tex_escape(rows_by.capitalize()) + " & " + " & ".join(
|
||||
tex_escape(disp(m)) + " " + arrow(d) for _, m, d in columns) + r" \\")
|
||||
else:
|
||||
m, d = specs[0]
|
||||
L.append(tex_escape(rows_by.capitalize()) + " & " + " & ".join(tex_escape(disp(c)) for c, _, _ in columns) + r" \\")
|
||||
L.append(r"\midrule")
|
||||
printed = []
|
||||
for r in row_vals:
|
||||
cells = []
|
||||
for c, m, d in columns:
|
||||
if (r, c, m) not in cell:
|
||||
cells.append("--"); continue
|
||||
mean, std, n = cell[(r, c, m)]
|
||||
txt = fmt(mean, digits)
|
||||
mk = marks.get((r, c, m))
|
||||
if mk == "best":
|
||||
txt = rf"\textbf{{{txt}}}"
|
||||
elif mk == "second":
|
||||
txt = rf"\underline{{{txt}}}"
|
||||
if marks.get((r, c, m, "tie")):
|
||||
txt += r"$^\dagger$"
|
||||
if show_std and n > 1:
|
||||
txt += rf"\,{{\scriptsize$\pm${fmt(std, digits)}}}"
|
||||
cells.append(txt)
|
||||
printed.append(f"{r:>18s} | {c:>14s} | {m:>10s} | mean {fmt(mean, digits)} std {fmt(std, digits)} n={n}{' BEST' if mk == 'best' else ''}")
|
||||
name = tex_escape(disp(r))
|
||||
if r == opt.get("highlight"):
|
||||
name = rf"{name} (ours)" if "ours" not in disp(r).lower() else name
|
||||
L.append(r"\midrule") if row_vals.index(r) == len(row_vals) - 1 else None
|
||||
L.append(name + " & " + " & ".join(cells) + r" \\")
|
||||
L += [r"\bottomrule", r"\end{tabular}", "}" if ncol > 4 else "", r"\end{table*}" if opt.get("wide") else r"\end{table}"]
|
||||
out = Path(opt.get("out", "paper/tables/main.tex"))
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text("\n".join(l for l in L if l != "") + "\n", encoding="utf-8")
|
||||
ns = sorted({v[2] for v in cell.values()})
|
||||
any_tie = any(len(k) == 4 for k in marks)
|
||||
seeds = str(ns[0]) if len(ns) == 1 else f"{ns[0]}--{ns[-1]}"
|
||||
note = "Mean $\\pm$ standard deviation over " + seeds + " seeds"
|
||||
if opt.get("scale"):
|
||||
note += " (values $\\times$" + str(opt["scale"]) + ")"
|
||||
note += ". Best in \\textbf{bold}, second \\underline{underlined}."
|
||||
if any_tie:
|
||||
note += " $^\\dagger$: not significantly worse than the best (Welch's $t$-test, $p \\geq 0.05$)."
|
||||
print("\n".join(printed))
|
||||
print(f"\nwrote {out} ({len(row_vals)} rows x {ncol} value columns)")
|
||||
print(f"caption note to append (edit caption text in the .tex via --caption):\n {note}")
|
||||
if max(ns) < 3:
|
||||
print("NOTE: < 3 seeds per cell - no significance marks possible; report this as a limitation.")
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- figures
|
||||
def mpl():
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
plt.rcParams.update({
|
||||
"font.family": "serif", "font.serif": ["Times New Roman", "Times", "Nimbus Roman", "DejaVu Serif"],
|
||||
"mathtext.fontset": "stix", "font.size": 8, "axes.titlesize": 8, "axes.labelsize": 8,
|
||||
"legend.fontsize": 7, "xtick.labelsize": 7, "ytick.labelsize": 7, "axes.linewidth": 0.6,
|
||||
"axes.spines.top": False, "axes.spines.right": False, "axes.grid": True, "grid.color": "#e3e3e3",
|
||||
"grid.linewidth": 0.5, "lines.linewidth": 1.4, "lines.markersize": 4, "legend.frameon": False,
|
||||
"savefig.bbox": "tight", "savefig.pad_inches": 0.02, "pdf.fonttype": 42, "ps.fonttype": 42})
|
||||
return plt
|
||||
|
||||
|
||||
def method_styles(methods, highlight=None):
|
||||
"""Stable colour/marker per method across all figures (persisted in style.json)."""
|
||||
try:
|
||||
style = json.loads(STYLE_FILE.read_text())
|
||||
except (OSError, json.JSONDecodeError):
|
||||
style = {}
|
||||
used = len(style)
|
||||
order = ([highlight] if highlight in methods and highlight not in style else []) + [m for m in methods if m not in style and m != highlight]
|
||||
for m in order:
|
||||
i = used
|
||||
style[m] = {"color": PALETTE[i % len(PALETTE)], "marker": MARKERS[i % len(MARKERS)],
|
||||
"ls": LINESTYLES[(i // len(PALETTE)) % len(LINESTYLES)]}
|
||||
used += 1
|
||||
STYLE_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
STYLE_FILE.write_text(json.dumps(style, indent=1))
|
||||
if len(methods) > len(PALETTE):
|
||||
print(f"NOTE: {len(methods)} methods > {len(PALETTE)} colours - colours repeat with other line styles/markers; "
|
||||
"consider fewer methods per figure (--where) or small multiples.")
|
||||
return style
|
||||
|
||||
|
||||
def figsize(opt, ncols=1):
|
||||
wopt = str(opt.get("width", ""))
|
||||
if re.fullmatch(r"\d+(\.\d+)?", wopt):
|
||||
w = float(wopt) # inches = the width the figure gets in the PDF (fonts stay 7-8 pt)
|
||||
else:
|
||||
w = 6.75 if wopt == "double" or ncols > 2 else 3.25
|
||||
h = float(opt.get("height", 2.0 if ncols <= 2 else 1.9))
|
||||
return w, h
|
||||
|
||||
|
||||
def facets(df, opt):
|
||||
f = opt.get("facet")
|
||||
if not f:
|
||||
return [(None, df)]
|
||||
if f not in df.columns:
|
||||
sys.exit(f"--facet {f}: no such column")
|
||||
return [(v, df[df[f] == v]) for v in ordered(df[f], csv_list(opt.get("facet-order")))]
|
||||
|
||||
|
||||
def save(fig, opt, used):
|
||||
out = Path(opt.get("out", "paper/figures/figure.pdf")).with_suffix(".pdf")
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
fig.savefig(out)
|
||||
fig.savefig(out.with_suffix(".png"), dpi=200)
|
||||
data = out.with_suffix(".data.csv")
|
||||
used.to_csv(data, index=False, float_format="%.6g")
|
||||
with pd.option_context("display.max_rows", 60, "display.width", 200):
|
||||
print(used.to_string(index=False, float_format=lambda v: f"{v:.4g}")[:3000])
|
||||
print(f"\nwrote {out} (+ .png preview for view_image, + {data.name} with the plotted numbers)")
|
||||
|
||||
|
||||
def legend(fig, axes, n):
|
||||
h, l = axes[0].get_legend_handles_labels()
|
||||
if n >= 2:
|
||||
fig.legend(h, l, loc="lower center", bbox_to_anchor=(0.5, 1.0), ncol=min(n, 4 if len(axes) == 1 else 6),
|
||||
handlelength=2.2, columnspacing=1.0)
|
||||
|
||||
|
||||
def cmd_figure(pos, opt):
|
||||
if len(pos) < 2:
|
||||
sys.exit("usage: figure line|bars|scatter results.csv ...")
|
||||
kind, path = pos[0], pos[1]
|
||||
df = load(path, opt.get("where"))
|
||||
scale = float(opt.get("scale", 1))
|
||||
plt = mpl()
|
||||
methods = ordered(df["method"], csv_list(opt.get("order")), opt.get("highlight"))
|
||||
style = method_styles(methods, opt.get("highlight"))
|
||||
groups = facets(df, opt)
|
||||
fig, axes = plt.subplots(1, len(groups), figsize=figsize(opt, len(groups)), squeeze=False)
|
||||
axes = list(axes[0])
|
||||
used_rows = []
|
||||
if kind == "line":
|
||||
metric, x = opt.get("metric"), opt.get("x", "step")
|
||||
if not metric or metric not in set(df["metric"]):
|
||||
sys.exit(f"--metric required; available: {sorted(df['metric'].unique())}")
|
||||
if x not in df.columns:
|
||||
sys.exit(f"--x {x}: no such column (columns: {list(df.columns)})")
|
||||
for ax, (fv, d) in zip(axes, groups):
|
||||
d = d[d["metric"] == metric].dropna(subset=["value", x])
|
||||
for m in methods:
|
||||
dm = d[d["method"] == m]
|
||||
if dm.empty:
|
||||
continue
|
||||
if x != "step" and "step" in dm.columns: # e.g. x = train_size: use the final value per run
|
||||
dm = final_values(dm, opt.get("final", "last"))
|
||||
s = dm.groupby(x)["value"].agg(["mean", "std", "count"]).reset_index().sort_values(x)
|
||||
s[["mean", "std"]] *= scale
|
||||
s["std"] = s["std"].fillna(0)
|
||||
st = style[m]
|
||||
ax.plot(s[x], s["mean"], color=st["color"], ls=st["ls"], marker=st["marker"],
|
||||
markevery=max(1, len(s) // 8), label=disp(m), zorder=3 if m == opt.get("highlight") else 2,
|
||||
lw=1.8 if m == opt.get("highlight") else 1.3)
|
||||
if (s["count"] > 1).any():
|
||||
ax.fill_between(s[x], s["mean"] - s["std"], s["mean"] + s["std"], color=st["color"], alpha=0.15, lw=0)
|
||||
used_rows.append(s.assign(method=m, facet=fv))
|
||||
ax.set_xlabel(opt.get("xlabel", disp(x).replace("_", " ")))
|
||||
if fv is not None:
|
||||
ax.set_title(disp(fv))
|
||||
axes[0].set_ylabel(opt.get("ylabel", disp(metric).replace("_", " ")))
|
||||
elif kind == "bars":
|
||||
metric = opt.get("metric")
|
||||
if not metric or metric not in set(df["metric"]):
|
||||
sys.exit(f"--metric required; available: {sorted(df['metric'].unique())}")
|
||||
fin = final_values(df[df["metric"] == metric], opt.get("final", "last"))
|
||||
fin["value"] *= scale
|
||||
group = opt.get("group", "dataset")
|
||||
for ax, (fv, d) in zip(axes, groups if opt.get("facet") else [(None, fin)]):
|
||||
if opt.get("facet"):
|
||||
d = fin[fin[opt["facet"]] == fv]
|
||||
gvals = ordered(d[group], csv_list(opt.get("group-order")))
|
||||
ms = [m for m in methods if m in set(d["method"])]
|
||||
width = 0.8 / max(1, len(ms))
|
||||
for j, m in enumerate(ms):
|
||||
s = d[d["method"] == m].groupby(group)["value"].agg(["mean", "std", "count"]).reindex(gvals)
|
||||
xs = np.arange(len(gvals)) + (j - (len(ms) - 1) / 2) * width
|
||||
st = style[m]
|
||||
ax.bar(xs, s["mean"], width * 0.92, yerr=s["std"].fillna(0) if (s["count"] > 1).any() else None,
|
||||
color=st["color"], label=disp(m), error_kw={"lw": 0.7, "capsize": 1.5, "ecolor": "#333333"}, zorder=2)
|
||||
used_rows.append(s.reset_index().assign(method=m, facet=fv))
|
||||
ax.set_xticks(range(len(gvals)))
|
||||
ax.set_xticklabels([disp(g) for g in gvals], rotation=0 if len(gvals) <= 4 else 30, ha="center" if len(gvals) <= 4 else "right")
|
||||
ax.grid(axis="x", visible=False)
|
||||
lo = d["value"].min()
|
||||
if lo > 0 and not opt.get("zero"):
|
||||
ax.set_ylim(bottom=0)
|
||||
if fv is not None:
|
||||
ax.set_title(disp(fv))
|
||||
axes[0].set_ylabel(opt.get("ylabel", disp(metric).replace("_", " ")))
|
||||
elif kind == "scatter":
|
||||
xm, ym = opt.get("x-metric"), opt.get("y-metric")
|
||||
avail = set(df["metric"])
|
||||
if xm not in avail or ym not in avail:
|
||||
sys.exit(f"--x-metric/--y-metric must be metrics; available: {sorted(avail)}")
|
||||
fin = final_values(df[df["metric"].isin([xm, ym])], opt.get("final", "last"))
|
||||
for ax, (fv, _) in zip(axes, groups):
|
||||
d = fin if fv is None else fin[fin[opt["facet"]] == fv]
|
||||
for m in methods:
|
||||
dm = d[d["method"] == m]
|
||||
if dm.empty:
|
||||
continue
|
||||
a = dm[dm["metric"] == xm]["value"]; b = dm[dm["metric"] == ym]["value"] * scale
|
||||
st = style[m]
|
||||
ax.errorbar(a.mean(), b.mean(), xerr=a.std() if len(a) > 1 else None, yerr=b.std() if len(b) > 1 else None,
|
||||
fmt=st["marker"], color=st["color"], ms=5, elinewidth=0.7, capsize=1.5, label=disp(m),
|
||||
mec="white", mew=0.6, zorder=3)
|
||||
used_rows.append(pd.DataFrame([{"method": m, "facet": fv, xm: a.mean(), ym: b.mean(),
|
||||
f"{xm}_std": a.std(), f"{ym}_std": b.std(), "n": len(b)}]))
|
||||
ax.set_xlabel(opt.get("xlabel", disp(xm).replace("_", " ")))
|
||||
if fv is not None:
|
||||
ax.set_title(disp(fv))
|
||||
axes[0].set_ylabel(opt.get("ylabel", disp(ym).replace("_", " ")))
|
||||
else:
|
||||
sys.exit("figure kinds: line, bars, scatter")
|
||||
for ax in axes:
|
||||
if opt.get("logx"): ax.set_xscale("log")
|
||||
if opt.get("logy"): ax.set_yscale("log")
|
||||
if opt.get("ylim"): ax.set_ylim(*[float(v) for v in opt["ylim"].split(",")])
|
||||
if opt.get("xlim"): ax.set_xlim(*[float(v) for v in opt["xlim"].split(",")])
|
||||
shown = len({r["method"].iloc[0] for r in used_rows if len(r)})
|
||||
legend(fig, axes, shown)
|
||||
fig.tight_layout()
|
||||
save(fig, opt, pd.concat(used_rows, ignore_index=True) if used_rows else pd.DataFrame())
|
||||
|
||||
|
||||
def main(argv):
|
||||
if not argv or argv[0] in ("-h", "--help"):
|
||||
print(__doc__); return 0
|
||||
cmd, (pos, opt) = argv[0], parse(argv[1:])
|
||||
RENAME.update(dict(kv.split("=", 1) for kv in csv_list(opt.get("rename")) if "=" in kv))
|
||||
fn = {"melt": cmd_melt, "tb": cmd_tb, "check": cmd_check, "summary": cmd_summary,
|
||||
"table": cmd_table, "figure": cmd_figure}.get(cmd)
|
||||
if not fn or (not pos and cmd != "figure"):
|
||||
print(__doc__); return 2
|
||||
fn(pos, opt)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
|
|
@ -0,0 +1,298 @@
|
|||
#!/usr/bin/env python3
|
||||
"""scan_experiments.py - inventory of the machine-learning experiments in a project (stdlib only).
|
||||
|
||||
python scan_experiments.py [ROOT] [--out paper/inventory]
|
||||
|
||||
Walks ROOT (default '.'), skipping paper/, .git, venvs, node_modules, caches, and classifies:
|
||||
code .py / .ipynb files that import an ML framework; entry points (__main__, argparse,
|
||||
hydra, click), CLI arguments with defaults, seeds, datasets, models, optimizers, metrics
|
||||
configs .yaml/.yml/.json/.toml files that look like hyperparameter configs
|
||||
results .csv/.tsv/.json/.jsonl result tables, *.log files with metric lines, TensorBoard
|
||||
event files, wandb/ and mlruns/ folders, checkpoints (only counted)
|
||||
docs README*, *.md, requirements.txt, environment.yml, pyproject.toml (dependencies)
|
||||
Writes <out>.json (full detail) and <out>.md (human-readable), prints a compact summary.
|
||||
The script only reads; nothing is executed.
|
||||
"""
|
||||
import ast, json, re, sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
SKIP = {"paper", ".git", "__pycache__", ".venv", "venv", "env", "node_modules", ".ipynb_checkpoints",
|
||||
".pytest_cache", ".mypy_cache", ".ruff_cache", "dist", "build", "site-packages", ".idea", ".vscode"}
|
||||
ML_LIBS = {"torch": "PyTorch", "tensorflow": "TensorFlow", "keras": "Keras", "jax": "JAX", "flax": "Flax",
|
||||
"sklearn": "scikit-learn", "xgboost": "XGBoost", "lightgbm": "LightGBM", "catboost": "CatBoost",
|
||||
"transformers": "Hugging Face Transformers", "datasets": "Hugging Face Datasets",
|
||||
"lightning": "PyTorch Lightning", "pytorch_lightning": "PyTorch Lightning",
|
||||
"torchvision": "torchvision", "timm": "timm", "peft": "PEFT", "trl": "TRL", "accelerate": "Accelerate",
|
||||
"optuna": "Optuna", "statsmodels": "statsmodels", "gymnasium": "Gymnasium", "gym": "Gym",
|
||||
"stable_baselines3": "Stable-Baselines3", "torch_geometric": "PyG", "dgl": "DGL",
|
||||
"sentence_transformers": "Sentence-Transformers", "wandb": "Weights & Biases", "mlflow": "MLflow",
|
||||
"numpy": "NumPy", "pandas": "pandas", "scipy": "SciPy"}
|
||||
CORE_ML = set(ML_LIBS) - {"numpy", "pandas", "scipy", "wandb", "mlflow", "optuna"}
|
||||
DATASETS = r"""MNIST|Fashion-?MNIST|KMNIST|EMNIST|CIFAR-?10|CIFAR-?100|SVHN|STL-?10|ImageNet(?:-1k|-21k)?|Tiny-?ImageNet|
|
||||
CelebA|LSUN|COCO|MS-?COCO|Pascal\s?VOC|Cityscapes|ADE20K|Places365|Caltech-?101|Caltech-?256|Oxford-?Pets|Flowers-?102|Food-?101|
|
||||
IMDB|SST-?2|GLUE|SuperGLUE|SQuAD|MNLI|QNLI|QQP|RTE|CoLA|AG-?News|Yelp|Amazon|WikiText-?\d*|Penn\s?Treebank|PTB|OpenWebText|
|
||||
C4|The\s?Pile|LAMBADA|HellaSwag|MMLU|GSM8K|HumanEval|MBPP|ARC|TruthfulQA|WMT\d*|Multi30k|LibriSpeech|Common\s?Voice|
|
||||
UCI|Adult|Iris|Wine|Breast\s?Cancer|California\s?Housing|Boston|Covertype|Higgs|Titanic|OpenML|
|
||||
Cora|CiteSeer|PubMed|OGB[\w-]*|QM9|ZINC|MovieLens|Criteo|ETTh\d|ETTm\d|Electricity|Traffic|Weather|M4|
|
||||
CartPole|MountainCar|Atari|MuJoCo|HalfCheetah|Hopper|Walker2d|Ant|Humanoid|Procgen|D4RL"""
|
||||
DATASET_RE = re.compile(r"(?<![\w-])(" + DATASETS.replace("\n", "") + r")(?![\w-])", re.I)
|
||||
HF_DS_RE = re.compile(r"load_dataset\(\s*['\"]([^'\"]+)['\"](?:\s*,\s*['\"]([^'\"]+)['\"])?")
|
||||
MODEL_RE = re.compile(r"(?<![\w])(ResNet-?\d+|ResNeXt-?\d*|WideResNet[\w-]*|WRN-?\d+-?\d*|VGG-?\d+|DenseNet-?\d+|"
|
||||
r"MobileNet(?:V\d)?|EfficientNet-?\w*|ViT-?[\w/]*|DeiT[\w-]*|Swin[\w-]*|ConvNeXt[\w-]*|U-?Net|"
|
||||
r"BERT[\w-]*|RoBERTa[\w-]*|DistilBERT|ALBERT|DeBERTa[\w-]*|ELECTRA|T5[\w-]*|BART|GPT-?\w*|"
|
||||
r"Llama[\w.-]*|Mistral[\w-]*|Qwen[\w.-]*|Gemma[\w-]*|Phi-?\d[\w-]*|CLIP|LSTM|GRU|Transformer|"
|
||||
r"MLP|CNN|GCN|GAT|GraphSAGE|GIN|VAE|GAN|DCGAN|StyleGAN\d?|DDPM|DDIM|Diffusion|"
|
||||
r"RandomForest\w*|GradientBoosting\w*|LogisticRegression|SVC|SVM|KNeighbors\w*|XGB\w*|LGBM\w*|"
|
||||
r"PPO|SAC|DQN|A2C|TD3|DDPG)(?![\w])", re.I)
|
||||
OPTIM_RE = re.compile(r"\b(SGD|Adam|AdamW|RMSprop|Adagrad|Adafactor|LAMB|LARS|Lion)\b")
|
||||
METRIC_WORDS = r"acc(?:uracy)?|top-?[15]|loss|f1|precision|recall|auc|auroc|auprc|mse|rmse|mae|r2|bleu|rouge[\w-]*|" \
|
||||
r"perplexity|ppl|map|miou|iou|dice|psnr|ssim|fid|is_score|reward|return|ece|nll|wer|cer|em|exact_match"
|
||||
METRIC_RE = re.compile(r"(?<![a-z])(" + METRIC_WORDS + r")(?![a-z])", re.I)
|
||||
SEED_RE = re.compile(r"(?:manual_seed|random\.seed|np\.random\.seed|seed_everything|set_seed|random_state)\s*[(=]\s*([\w.]+)")
|
||||
LOG_METRIC_RE = re.compile(r"(?i)\b(" + METRIC_WORDS + r"|val_\w+|test_\w+|train_\w+)\s*[:=]\s*(-?\d+\.?\d*(?:e-?\d+)?)")
|
||||
HP_KEYS = {"lr", "learning_rate", "batch_size", "epochs", "num_epochs", "max_epochs", "weight_decay", "momentum",
|
||||
"dropout", "hidden_size", "hidden_dim", "num_layers", "seed", "optimizer", "warmup", "max_steps",
|
||||
"n_estimators", "max_depth", "model", "dataset", "scheduler", "temperature", "gamma", "beta1", "beta2"}
|
||||
RESULT_HINT = re.compile(r"(?i)result|metric|score|eval|summary|log|history|progress|benchmark|runs?|ablation|output")
|
||||
|
||||
|
||||
def walk(root):
|
||||
for p in sorted(root.rglob("*")):
|
||||
rel = p.relative_to(root).parts
|
||||
if any(part in SKIP or (part.startswith(".") and part != ".") for part in rel[:-1]):
|
||||
continue
|
||||
if p.is_file():
|
||||
yield p
|
||||
|
||||
|
||||
def read(p, limit=2_000_000):
|
||||
try:
|
||||
return p.read_text(encoding="utf-8", errors="replace")[:limit]
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def py_info(src, name):
|
||||
info = {"imports": set(), "entry_point": False, "cli_args": [], "seeds": [], "functions": [], "classes": []}
|
||||
try:
|
||||
tree = ast.parse(src)
|
||||
except SyntaxError:
|
||||
tree = None
|
||||
if tree:
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Import):
|
||||
info["imports"] |= {a.name.split(".")[0] for a in node.names}
|
||||
elif isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
|
||||
info["imports"].add(node.module.split(".")[0])
|
||||
elif isinstance(node, ast.FunctionDef) and node.col_offset == 0:
|
||||
info["functions"].append(node.name)
|
||||
elif isinstance(node, ast.ClassDef) and node.col_offset == 0:
|
||||
bases = [ast.unparse(b) for b in node.bases]
|
||||
info["classes"].append(f"{node.name}({', '.join(bases)})" if bases else node.name)
|
||||
elif isinstance(node, ast.Call) and getattr(node.func, "attr", "") == "add_argument":
|
||||
flags = [a.value for a in node.args if isinstance(a, ast.Constant) and isinstance(a.value, str)]
|
||||
kw = {k.arg: ast.unparse(k.value) for k in node.keywords if k.arg in ("default", "type", "help", "choices")}
|
||||
if flags:
|
||||
info["cli_args"].append({"flag": flags[-1], **{k: v[:80] for k, v in kw.items()}})
|
||||
info["entry_point"] = bool(re.search(r"__name__\s*==\s*['\"]__main__['\"]|@hydra\.main|@click\.command|typer\.run", src))
|
||||
info["seeds"] = sorted(set(SEED_RE.findall(src)))[:10]
|
||||
return info
|
||||
|
||||
|
||||
def nb_source(text):
|
||||
try:
|
||||
nb = json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
return "", []
|
||||
code, outs = [], []
|
||||
for c in nb.get("cells", []):
|
||||
s = "".join(c.get("source", []))
|
||||
if c.get("cell_type") == "code":
|
||||
code.append(s)
|
||||
for o in c.get("outputs", []):
|
||||
t = "".join(o.get("text", []) or o.get("data", {}).get("text/plain", []))
|
||||
if t and LOG_METRIC_RE.search(t):
|
||||
outs.append(t.strip()[:300])
|
||||
elif c.get("cell_type") == "markdown":
|
||||
code.append("# " + s.replace("\n", "\n# "))
|
||||
return "\n".join(code), outs[:10]
|
||||
|
||||
|
||||
def table_info(p, text):
|
||||
if p.suffix in (".csv", ".tsv"):
|
||||
lines = [l for l in text.splitlines() if l.strip()]
|
||||
if not lines:
|
||||
return None
|
||||
sep = "\t" if p.suffix == ".tsv" or lines[0].count("\t") > lines[0].count(",") else ","
|
||||
return {"kind": "table", "columns": [c.strip() for c in lines[0].split(sep)][:40], "rows": len(lines) - 1,
|
||||
"head": lines[1:4]}
|
||||
if p.suffix == ".jsonl":
|
||||
lines = text.splitlines()
|
||||
try:
|
||||
first = json.loads(lines[0])
|
||||
except (json.JSONDecodeError, IndexError):
|
||||
return None
|
||||
return {"kind": "jsonl", "keys": list(first)[:40] if isinstance(first, dict) else None, "rows": len(lines),
|
||||
"head": lines[:2]}
|
||||
if p.suffix == ".json":
|
||||
try:
|
||||
js = json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
def shape(x, d=0):
|
||||
if d > 2: return "..."
|
||||
if isinstance(x, dict): return {k: shape(v, d + 1) for k, v in list(x.items())[:15]}
|
||||
if isinstance(x, list): return [f"list[{len(x)}]", shape(x[0], d + 1) if x else None]
|
||||
return type(x).__name__ if not isinstance(x, (int, float)) else x
|
||||
return {"kind": "json", "shape": shape(js)}
|
||||
return None
|
||||
|
||||
|
||||
def is_config(text, suffix):
|
||||
keys = set(re.findall(r"(?m)^\s*[\"']?([A-Za-z_][\w.-]*)[\"']?\s*[:=]", text))
|
||||
return len({k.lower().split(".")[-1] for k in keys} & HP_KEYS) >= 2
|
||||
|
||||
|
||||
def scan(root):
|
||||
inv = {"root": str(root.resolve()), "code": [], "notebooks": [], "configs": [], "results": [], "logs": [],
|
||||
"tensorboard": [], "trackers": [], "checkpoints": Counter(), "docs": [], "figures": [],
|
||||
"libraries": Counter(), "datasets": Counter(), "models": Counter(), "optimizers": Counter(),
|
||||
"metrics": Counter()}
|
||||
for d in ("wandb", "mlruns", "lightning_logs", "runs", "outputs", "multirun"):
|
||||
for p in root.rglob(d):
|
||||
if p.is_dir() and not (set(p.relative_to(root).parts) & SKIP):
|
||||
inv["trackers"].append(str(p.relative_to(root)))
|
||||
for p in walk(root):
|
||||
rel, suf, name = str(p.relative_to(root)), p.suffix.lower(), p.name.lower()
|
||||
if suf in (".pt", ".pth", ".ckpt", ".safetensors", ".h5", ".keras", ".pkl", ".joblib", ".onnx", ".bin", ".npz", ".npy"):
|
||||
inv["checkpoints"][suf] += 1; continue
|
||||
if "tfevents" in name:
|
||||
inv["tensorboard"].append(rel); continue
|
||||
if suf in (".png", ".jpg", ".jpeg", ".pdf", ".svg"):
|
||||
inv["figures"].append(rel); continue
|
||||
if name.startswith("readme") or name in ("requirements.txt", "environment.yml", "pyproject.toml", "setup.py", "agents.md") or suf == ".md":
|
||||
text = read(p, 20_000)
|
||||
inv["docs"].append({"path": rel, "first_lines": [l for l in text.splitlines() if l.strip()][:6]})
|
||||
inv["datasets"].update(m.group(1) for m in DATASET_RE.finditer(text))
|
||||
inv["models"].update(MODEL_RE.findall(text))
|
||||
continue
|
||||
if suf in (".py", ".ipynb"):
|
||||
text = read(p)
|
||||
outs = []
|
||||
if suf == ".ipynb":
|
||||
text, outs = nb_source(text)
|
||||
info = py_info(text, rel)
|
||||
libs = sorted(i for i in info["imports"] if i in ML_LIBS)
|
||||
inv["libraries"].update(ML_LIBS[l] for l in libs)
|
||||
ds = Counter(m.group(1) for m in DATASET_RE.finditer(text))
|
||||
ds.update(":".join(x for x in m.groups() if x) for m in HF_DS_RE.finditer(text))
|
||||
models = Counter(MODEL_RE.findall(text))
|
||||
optims = Counter(OPTIM_RE.findall(text))
|
||||
metrics = Counter(m.lower() for m in METRIC_RE.findall(text))
|
||||
inv["datasets"].update(ds); inv["models"].update(models); inv["optimizers"].update(optims)
|
||||
if not (set(libs) & CORE_ML) and not ds and not info["entry_point"]:
|
||||
continue # plain helper without ML content
|
||||
inv["metrics"].update(metrics)
|
||||
entry = {"path": rel, "lines": text.count("\n") + 1, "ml_libraries": [ML_LIBS[l] for l in libs],
|
||||
"entry_point": info["entry_point"], "cli_args": info["cli_args"][:40], "seeds": info["seeds"],
|
||||
"datasets": sorted(ds), "models": sorted(models), "optimizers": sorted(optims),
|
||||
"metrics": sorted(metrics), "functions": info["functions"][:30], "classes": info["classes"][:20]}
|
||||
if outs:
|
||||
entry["metric_outputs"] = outs
|
||||
inv["notebooks" if suf == ".ipynb" else "code"].append(entry)
|
||||
continue
|
||||
if suf in (".yaml", ".yml", ".toml", ".cfg", ".ini") or (suf == ".json" and not RESULT_HINT.search(rel)):
|
||||
text = read(p, 200_000)
|
||||
if is_config(text, suf):
|
||||
inv["configs"].append({"path": rel, "preview": [l for l in text.splitlines() if l.strip()][:25]})
|
||||
inv["datasets"].update(m.group(1) for m in DATASET_RE.finditer(text))
|
||||
continue
|
||||
if suf != ".json":
|
||||
continue
|
||||
if suf in (".csv", ".tsv", ".json", ".jsonl"):
|
||||
text = read(p)
|
||||
ti = table_info(p, text)
|
||||
if ti:
|
||||
cols = " ".join(ti.get("columns") or ti.get("keys") or [str(ti.get("shape"))[:500]])
|
||||
inv["metrics"].update(m.lower() for m in METRIC_RE.findall(cols))
|
||||
inv["results"].append({"path": rel, "bytes": p.stat().st_size, **ti})
|
||||
continue
|
||||
if suf in (".log", ".txt", ".out", ".err"):
|
||||
text = read(p)
|
||||
hits = LOG_METRIC_RE.findall(text)
|
||||
if hits:
|
||||
inv["logs"].append({"path": rel, "metric_lines": len(hits),
|
||||
"metrics": sorted({h[0].lower() for h in hits})[:20],
|
||||
"last": [l for l in text.splitlines() if LOG_METRIC_RE.search(l)][-3:]})
|
||||
for k in ("libraries", "datasets", "models", "optimizers", "metrics", "checkpoints"):
|
||||
inv[k] = dict(inv[k].most_common(40))
|
||||
return inv
|
||||
|
||||
|
||||
def to_md(inv):
|
||||
L = [f"# Experiment inventory\n\nRoot: `{inv['root']}`\n"]
|
||||
for k in ("libraries", "datasets", "models", "optimizers", "metrics"):
|
||||
L.append(f"- **{k}**: " + (", ".join(f"{a} ({n})" for a, n in inv[k].items()) or "-"))
|
||||
L.append(f"- **checkpoints**: {inv['checkpoints'] or '-'} | **tensorboard files**: {len(inv['tensorboard'])}"
|
||||
f" | **trackers**: {', '.join(inv['trackers']) or '-'}\n")
|
||||
L.append("## Code (entry points first)\n")
|
||||
for c in sorted(inv["code"] + inv["notebooks"], key=lambda c: (not c["entry_point"], c["path"])):
|
||||
L.append(f"### `{c['path']}` ({c['lines']} lines{', ENTRY POINT' if c['entry_point'] else ''})")
|
||||
L.append(f"libs: {', '.join(c['ml_libraries']) or '-'}; datasets: {', '.join(c['datasets']) or '-'}; "
|
||||
f"models: {', '.join(c['models']) or '-'}; optimizers: {', '.join(c['optimizers']) or '-'}; "
|
||||
f"metrics: {', '.join(c['metrics']) or '-'}; seeds: {', '.join(c['seeds']) or '-'}")
|
||||
if c["cli_args"]:
|
||||
L.append("args: " + "; ".join(f"{a['flag']}={a.get('default', '?')}" for a in c["cli_args"]))
|
||||
if c["classes"]:
|
||||
L.append("classes: " + ", ".join(c["classes"]))
|
||||
if c["functions"]:
|
||||
L.append("functions: " + ", ".join(c["functions"]))
|
||||
for o in c.get("metric_outputs", []):
|
||||
L.append(f" output> {o[:200]}")
|
||||
L.append("")
|
||||
L.append("## Configs\n")
|
||||
for c in inv["configs"]:
|
||||
L.append(f"### `{c['path']}`\n```\n" + "\n".join(c["preview"]) + "\n```")
|
||||
L.append("\n## Result files\n")
|
||||
for r in inv["results"]:
|
||||
desc = (f"columns={r['columns']}, rows={r['rows']}" if r["kind"] == "table" else
|
||||
f"keys={r['keys']}, rows={r['rows']}" if r["kind"] == "jsonl" else f"shape={json.dumps(r['shape'])[:400]}")
|
||||
L.append(f"- `{r['path']}` ({r['kind']}, {r['bytes']} B): {desc}")
|
||||
L.append("\n## Logs with metrics\n")
|
||||
for g in inv["logs"]:
|
||||
L.append(f"- `{g['path']}`: {g['metric_lines']} metric lines, {g['metrics']}; last: {g['last'][-1][:150] if g['last'] else ''}")
|
||||
if inv["tensorboard"]:
|
||||
L.append("\n## TensorBoard event files\n" + "\n".join(f"- `{t}`" for t in inv["tensorboard"][:50]))
|
||||
if inv["figures"]:
|
||||
L.append("\n## Existing figures (not reused blindly - regenerate from data)\n" + "\n".join(f"- `{f}`" for f in inv["figures"][:50]))
|
||||
L.append("\n## Docs\n")
|
||||
for d in inv["docs"]:
|
||||
L.append(f"- `{d['path']}`: " + " / ".join(d["first_lines"])[:300])
|
||||
return "\n".join(L) + "\n"
|
||||
|
||||
|
||||
def main(argv):
|
||||
if "-h" in argv or "--help" in argv:
|
||||
print(__doc__); return 0
|
||||
out = Path(argv[argv.index("--out") + 1]) if "--out" in argv else Path("paper/inventory")
|
||||
pos = [a for i, a in enumerate(argv) if not a.startswith("--") and (i == 0 or argv[i - 1] != "--out")]
|
||||
root = Path(pos[0] if pos else ".")
|
||||
inv = scan(root)
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.with_suffix(".json").write_text(json.dumps(inv, indent=1, ensure_ascii=False, default=list), encoding="utf-8")
|
||||
out.with_suffix(".md").write_text(to_md(inv), encoding="utf-8")
|
||||
entries = [c["path"] for c in inv["code"] + inv["notebooks"] if c["entry_point"]]
|
||||
print(f"wrote {out}.md and {out}.json")
|
||||
print(f"code files with ML content: {len(inv['code'])} (+{len(inv['notebooks'])} notebooks); entry points: {', '.join(entries[:12]) or '-'}")
|
||||
print(f"configs: {len(inv['configs'])}; result files: {len(inv['results'])}; logs with metrics: {len(inv['logs'])}; "
|
||||
f"tensorboard: {len(inv['tensorboard'])}; trackers: {', '.join(inv['trackers']) or '-'}")
|
||||
for k in ("libraries", "datasets", "models", "metrics"):
|
||||
print(f"{k}: " + (", ".join(list(inv[k])[:12]) or "-"))
|
||||
if not inv["results"] and not inv["logs"] and not inv["tensorboard"]:
|
||||
print("NOTE: no result files found - the experiments may not have been run yet.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
% main.tex - generic ML conference paper (single column, ~NeurIPS/ICLR geometry, natbib author-year).
|
||||
% For a specific venue: replace the \documentclass/geometry block with the official style file
|
||||
% (e.g. \usepackage{neurips_2025}, \usepackage{icml2025}, \documentclass{llncs}) and keep the rest.
|
||||
\documentclass[11pt]{article}
|
||||
\usepackage[letterpaper, textwidth=5.5in, textheight=9in, top=1in]{geometry}
|
||||
|
||||
\usepackage[utf8]{inputenc}
|
||||
% T1 + Latin Modern when installed (scalable fonts); otherwise the default Computer Modern
|
||||
\IfFileExists{lmodern.sty}{\usepackage[T1]{fontenc}\usepackage{lmodern}}{}
|
||||
\usepackage{microtype}
|
||||
\usepackage{amsmath, amssymb, amsthm, mathtools}
|
||||
\usepackage{bm}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{multirow}
|
||||
\usepackage{subcaption}
|
||||
\usepackage{xcolor}
|
||||
\usepackage[numbers, sort&compress]{natbib} % for author-year: \usepackage[round]{natbib}
|
||||
\usepackage[colorlinks=true, linkcolor=blue!60!black, citecolor=blue!60!black, urlcolor=blue!60!black]{hyperref}
|
||||
\usepackage[capitalise, noabbrev]{cleveref}
|
||||
\usepackage{xspace}
|
||||
\usepackage{algorithm}
|
||||
\usepackage{algpseudocode}
|
||||
|
||||
% ---- macros: one name for the proposed method everywhere ------------------
|
||||
\newcommand{\ours}{\textsc{MethodName}}
|
||||
\newcommand{\eg}{e.g.,\xspace}
|
||||
\newcommand{\ie}{i.e.,\xspace}
|
||||
\newcommand{\etal}{et al.\xspace}
|
||||
\newcommand{\R}{\mathbb{R}}
|
||||
\newcommand{\E}{\mathbb{E}}
|
||||
\DeclareMathOperator*{\argmin}{arg\,min}
|
||||
\DeclareMathOperator*{\argmax}{arg\,max}
|
||||
\newtheorem{proposition}{Proposition}
|
||||
|
||||
\title{Paper Title: Specific, Informative, Without Hype}
|
||||
% Double-blind by default. Fill in real authors only for camera-ready / arXiv.
|
||||
\author{Anonymous Authors\\Paper under double-blind review}
|
||||
\date{}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
\input{sections/abstract}
|
||||
\end{abstract}
|
||||
|
||||
\input{sections/introduction}
|
||||
\input{sections/related_work}
|
||||
\input{sections/method}
|
||||
\input{sections/experimental_setup}
|
||||
\input{sections/results}
|
||||
\input{sections/discussion}
|
||||
\input{sections/conclusion}
|
||||
|
||||
% Statements many venues require (do not count towards the page limit at most venues)
|
||||
\input{sections/statements}
|
||||
|
||||
\bibliographystyle{plainnat}
|
||||
\bibliography{references}
|
||||
|
||||
\appendix
|
||||
\input{sections/appendix}
|
||||
|
||||
\end{document}
|
||||
|
|
@ -0,0 +1,107 @@
|
|||
# Writing guide: section by section
|
||||
|
||||
Read the section you are about to write, then write it into `paper/sections/<name>.tex`.
|
||||
Every section is plain LaTeX without `\section` preamble tricks; cite with `\citep{key}` /
|
||||
`\citet{key}` (natbib), reference with `\cref{fig:x}` / `\cref{tab:x}` / `\cref{sec:x}`.
|
||||
|
||||
General style
|
||||
- One claim per sentence; every claim is backed by (a) a table/figure of this paper,
|
||||
(b) a citation, or (c) a derivation. If none exists, delete the claim.
|
||||
- Use the method name macro `\ours{}` consistently. Define every symbol once, at first use.
|
||||
- Present tense for what the paper does ("we propose", "Table 2 shows"), past tense for
|
||||
what was run ("we trained for 100 epochs").
|
||||
- No hype words: novel, groundbreaking, remarkable, significantly (unless a test was run),
|
||||
state-of-the-art (unless compared against the current best published number on the
|
||||
same benchmark with the same protocol, cited).
|
||||
- Numbers: same number of decimals as the table; units; "x pp" for percentage-point
|
||||
differences. Quote only values from `paper/data/summary.csv`, the generated tables, or a
|
||||
cited paper (then cite it in the same sentence).
|
||||
- Paragraphs open with a topic sentence; a reader who reads only first sentences gets
|
||||
the story.
|
||||
|
||||
## abstract.tex (150-250 words, no citations, no math)
|
||||
1. Context + problem (1-2 sentences). 2. Gap: why existing methods fall short (1).
|
||||
3. What we do: method in one sentence, the key idea (1-2). 4. Evidence: datasets and the
|
||||
headline result with a number (1-2). 5. Implication / what is released (1).
|
||||
|
||||
## introduction.tex (~1 page) - `\section{Introduction}\label{sec:intro}`
|
||||
- P1 Motivation: the task and why it matters, with 2-4 citations.
|
||||
- P2 The gap: what current approaches do and the specific limitation, cited.
|
||||
- P3 Our idea: the insight in plain words; optionally point to the teaser figure.
|
||||
- P4 Evidence: what was evaluated and the headline numbers.
|
||||
- Contributions as an `itemize` of 3-4 bullets, each concrete and checkable
|
||||
("We show on X and Y that ... improves Z by N pp over B", not "extensive experiments").
|
||||
- Optional last line: paper outline (skip if tight on space).
|
||||
|
||||
## related_work.tex (0.5-1 page) - `\section{Related Work}\label{sec:related}`
|
||||
- 2-4 `\paragraph{Theme.}` blocks, grouped by idea, not a list of papers.
|
||||
- Each paragraph: what the line of work does (cite 3-8 papers), then **how this paper
|
||||
differs** in one sentence. Never misrepresent a paper: describe only what its
|
||||
abstract/text (as summarised in research.md) says.
|
||||
- Cite the original paper of every baseline, dataset, model and major library used.
|
||||
|
||||
## method.tex (1.5-2.5 pages) - `\section{Method}\label{sec:method}`
|
||||
- Problem setting / notation first (`\subsection{Problem Setting}`): inputs, outputs,
|
||||
objective, as equations.
|
||||
- Then the method, in the order the code executes it. Every equation must match the code
|
||||
(loss terms, normalisations, hyperparameters as symbols). Read the code again while
|
||||
writing this; if the code differs from the "intended" method, describe the code.
|
||||
- An algorithm box (`algorithm` + `algorithmic`) if there is a training/inference loop.
|
||||
- Complexity / cost if relevant (params, FLOPs, wall-clock - only if measured).
|
||||
- Design choices: say why, ideally pointing to an ablation.
|
||||
|
||||
## experimental_setup.tex (~0.75 page) - `\section{Experimental Setup}\label{sec:setup}`
|
||||
Reproducibility details taken from the code/configs, not invented:
|
||||
- Datasets (cite each): size, splits, preprocessing, augmentation.
|
||||
- Baselines (cite each) and how they were tuned (same budget?).
|
||||
- Architecture, optimizer, learning rate + schedule, batch size, epochs/steps, weight
|
||||
decay, early stopping, seeds (number of runs), hardware and run time if logged.
|
||||
- Metrics, with direction (higher/lower is better) and how they are aggregated
|
||||
(mean +- std over N seeds; test-set at last epoch or at best validation epoch).
|
||||
- Put long hyperparameter tables in the appendix and reference them.
|
||||
|
||||
## results.tex (1.5-2.5 pages) - `\section{Results}\label{sec:results}`
|
||||
- `\subsection{Main Results}`: `\input{tables/main}` + 1 paragraph per finding. Each
|
||||
paragraph: observation -> numbers -> interpretation. Mention where the method does
|
||||
NOT win.
|
||||
- `\subsection{Ablations}` (if variant/ablation runs exist): what each component
|
||||
contributes, `\input{tables/ablation}` or a bar figure.
|
||||
- `\subsection{Analysis}`: learning curves, sensitivity to a hyperparameter, efficiency
|
||||
trade-off, qualitative examples - whatever the data supports.
|
||||
- Figures: `\begin{figure}[t]\centering\includegraphics[width=\linewidth]{figures/x.pdf}
|
||||
\caption{What is shown. What to notice. Shaded: +-1 std over N seeds.}\label{fig:x}\end{figure}`.
|
||||
Captions are self-contained; every figure/table is referenced in the text before it appears.
|
||||
- Significance: only call a difference significant if a test was run (the dagger marks
|
||||
in results.py tables); otherwise say "higher on average" and give std.
|
||||
|
||||
## discussion.tex (~0.5 page) - `\section{Discussion and Limitations}\label{sec:discussion}`
|
||||
- What the results mean, when to use the method, when not.
|
||||
- Limitations honestly and specifically: few seeds, small datasets, single domain, no
|
||||
hyperparameter search for baselines, compute limits, missing comparisons. Reviewers
|
||||
trust a paper more when it names its weaknesses.
|
||||
|
||||
## conclusion.tex (1 paragraph) - `\section{Conclusion}\label{sec:conclusion}`
|
||||
Restate the problem, the idea, the evidence; one sentence of future work. No new claims.
|
||||
|
||||
## statements.tex
|
||||
`\section*{Reproducibility Statement}`: where code/configs/seeds are, which sections/appendix
|
||||
give the details. `\section*{Broader Impact}` (or Ethics Statement): 3-6 honest sentences;
|
||||
dataset licences if relevant. Keep what the target venue requires, drop the rest.
|
||||
|
||||
## appendix.tex
|
||||
`\section{Additional Experimental Details}\label{app:details}` (full hyperparameter table,
|
||||
compute), `\section{Additional Results}` (per-dataset tables, more curves). Every appendix
|
||||
section is referenced from the main text.
|
||||
|
||||
## Title
|
||||
Specific and searchable: "<Method>: <what it does> for <task>" or a finding-style title
|
||||
("<X> improves <Y> under <condition>"). No question titles, no puns unless the user wants them.
|
||||
|
||||
## Self-review checklist (do it as a reviewer, write findings to paper/review.md)
|
||||
- Is the contribution clear after page 1? Are claims in abstract/intro all supported by results?
|
||||
- Are baselines fair (same data, budget, tuning)? Are the strongest relevant baselines there?
|
||||
- Seeds/variance reported? Test set used only for final numbers?
|
||||
- Are related works that do nearly the same thing cited and differentiated?
|
||||
- Do equations match the code? Notation consistent? All symbols defined?
|
||||
- Every figure readable in grayscale print, axes labelled with units, fonts >= 7pt?
|
||||
- Limitations stated? Anything over-claimed ("significantly", "state of the art")?
|
||||
Loading…
Reference in New Issue