171 lines
11 KiB
Markdown
171 lines
11 KiB
Markdown
---
|
|
name: write-my-paper
|
|
description: >
|
|
Write a complete machine-learning conference paper in LaTeX from the experiment code and
|
|
results in the project directory: inventory the experiments, collect the results, do a
|
|
scientific literature search with real citations, generate figures and tables from the data,
|
|
write all sections, compile to PDF and self-review. Use for "write my paper", "write a paper
|
|
about these experiments", "turn my results into a conference paper", "draft the paper".
|
|
---
|
|
|
|
# Write my paper
|
|
|
|
Goal: `paper/main.pdf`, built from `paper/main.tex` + `paper/sections/*.tex`, with figures and
|
|
tables generated from the actual results, and citations that all come from resolved records.
|
|
The project directory (your working directory, jcode's `./code`) holds the experiments.
|
|
Write ONLY inside `paper/`; never edit the experiment code.
|
|
|
|
## Non-negotiable rules
|
|
1. **No invented results.** Every number in the paper comes from `paper/data/summary.csv`, a
|
|
generated table/figure, or a cited paper (cited in the same sentence). `check_paper.py`
|
|
flags every decimal number it cannot trace - fix each one.
|
|
2. **No invented citations.** Every `\cite` key comes from the `scientific-deep-research` skill
|
|
(resolver + BibTeX), never from memory. Describe a cited paper only as far as you read it.
|
|
3. **No invented experiments.** Describe only what the code does and what was run. If the code
|
|
and the README disagree, the code wins. Missing evidence goes into Limitations, or is
|
|
reported to the user - not papered over.
|
|
4. **Honest framing.** Say where the method loses. "Significant" only with a test.
|
|
|
|
## Tools of this skill
|
|
`load_skill` prints this skill's directory; run the scripts from there with an absolute path,
|
|
e.g. `python <skill-dir>/results.py check paper/data/results.csv`. Use `timeout=600` for
|
|
`run_command` calls that compile LaTeX, search literature, or run experiments. Outputs are
|
|
truncated after 4000 characters: the scripts write full reports to files - read those with
|
|
`read_file` (in ranges).
|
|
- [scan_experiments.py](./scan_experiments.py) - inventory of code, configs, result files, logs,
|
|
datasets, models, metrics -> `paper/inventory.md` / `.json`.
|
|
- [results.py](./results.py) - tidy results CSV -> `check`, `summary`, booktabs `table`,
|
|
`figure line|bars|scatter` (PDF + PNG preview + the plotted numbers as CSV). `melt` converts
|
|
wide CSVs, `tb` reads TensorBoard logs. Run it without arguments for the full usage.
|
|
- [check_paper.py](./check_paper.py) - compile + errors, undefined refs/cites, placeholders,
|
|
number provenance, page count -> `paper/check_report.md`.
|
|
- [template/main.tex](./template/main.tex) - generic conference skeleton (natbib, cleveref).
|
|
- [writing-guide.md](./writing-guide.md) - what goes into every section + reviewer checklist.
|
|
Read it with `load_skill('write-my-paper', 'writing-guide.md')` before phase 6.
|
|
- Literature: the `scientific-deep-research` skill (`papers.py`).
|
|
Missing packages: `pip install pandas numpy matplotlib scipy pymupdf` (and `tensorboard` only
|
|
for TensorBoard logs). LaTeX needs `latexmk`/`pdflatex`+`bibtex` or `tectonic`; if none exists,
|
|
say so in the final summary and still deliver the .tex sources.
|
|
|
|
## Progress file
|
|
Keep `paper/STATUS.md` up to date after every phase: phase done, files produced, decisions,
|
|
open problems. If the context was compacted or the task is resumed, read `paper/STATUS.md`
|
|
first and continue from the first unfinished phase - don't redo finished phases.
|
|
|
|
## Procedure
|
|
|
|
### Phase 0 - Setup
|
|
`mkdir -p paper/data paper/figures paper/tables paper/sections paper/related`, copy the
|
|
template: `cp <skill-dir>/template/main.tex paper/`. Defaults unless the user said otherwise:
|
|
generic conference format, double-blind ("Anonymous Authors"), ~8-9 pages main text, English.
|
|
If the user named a venue, note its page limit and style in STATUS.md (swap the
|
|
`\documentclass`/geometry lines for the official style file if the user provides it).
|
|
|
|
### Phase 1 - Understand the experiments
|
|
1. `python <skill-dir>/scan_experiments.py . --out paper/inventory`, then read `paper/inventory.md`.
|
|
2. Read README/docs, then every entry-point script, the model/loss code, the configs, and the
|
|
head of each result file. Use `search_files` to locate loss functions, `argparse` defaults,
|
|
seeds, metric computations, data splits.
|
|
3. Write `paper/experiments.md`:
|
|
- research question and hypothesis (one sentence each);
|
|
- the proposed method ("ours") vs. baselines - what exactly is new, with file:line refs;
|
|
- datasets (+ splits, preprocessing), models, training setup (optimizer, lr, batch size,
|
|
epochs, seeds, hardware if logged), metrics + direction (higher/lower is better);
|
|
- experiment list: main comparison, ablations, sensitivity/scaling studies, with the
|
|
result file(s) of each and how many seeds exist;
|
|
- gaps: experiments the code supports but whose results are missing.
|
|
If "ours" vs. baseline cannot be determined from code/README/names, state your best
|
|
reading and mark it as an assumption in STATUS.md and the final summary.
|
|
4. Missing results: run an experiment only if it is clearly cheap (a few minutes). Longer runs:
|
|
start them detached (`nohup python train.py ... > paper/data/run_x.log 2>&1 &`) only if the
|
|
user asked for it; otherwise list the exact commands under "open problems" and continue with
|
|
the results that exist.
|
|
|
|
### Phase 2 - Results into one tidy table
|
|
1. Target: `paper/data/results.csv` with columns `method,dataset,metric,value[,seed][,step]`
|
|
plus extra columns as needed (e.g. `variant` for ablations, `train_size`, `noise`).
|
|
Method names exactly as they should appear in the paper (e.g. `ResNet-18 + Ours`).
|
|
2. Wide CSVs: `results.py melt <file> --id method,dataset,seed --out paper/data/results.csv
|
|
[--append]`. TensorBoard: `results.py tb <logdir> ...`. Anything else (JSON, logs, nested
|
|
folders): write `paper/data/extract.py` that reads the raw files and writes the tidy CSV -
|
|
keep it, it documents where every number came from.
|
|
3. `results.py check paper/data/results.csv` - fix duplicates, NaNs, wrong method names.
|
|
4. `results.py summary paper/data/results.csv [--final last|best:max|best:min]` ->
|
|
`paper/data/summary.csv`. `--final` must match the protocol in the code (last epoch vs.
|
|
best validation epoch). Note the choice in experiments.md.
|
|
|
|
### Phase 3 - Literature research
|
|
`load_skill('scientific-deep-research')` and follow it, working inside `paper/related/`
|
|
(`cd paper/related && python <research-skill-dir>/papers.py ...`). Search phrases must cover:
|
|
the task/problem, the method family of "ours", each baseline, each dataset, and a **novelty
|
|
check** (the core idea in plain words - is it already published? If yes: cite it and position
|
|
the paper honestly as a study/extension). Deep-read 10-15 core papers; additionally resolve +
|
|
BibTeX the original paper of every baseline, dataset, model architecture and main library
|
|
(PyTorch, scikit-learn, ...) used - for those, the abstract is enough. Aim for 25-45 references.
|
|
Result: `paper/related/research.md` and `paper/related/references.bib`;
|
|
`cp paper/related/references.bib paper/references.bib`.
|
|
|
|
### Phase 4 - Story and outline
|
|
Write `paper/outline.md`: working title; the one-sentence contribution; 3-4 contributions,
|
|
each mapped to its evidence (table/figure + the numbers from summary.csv); the planned
|
|
figures/tables (usually: 1 main results table, 1 ablation table or bar chart, 1-2 analysis
|
|
figures such as learning curves / sensitivity / accuracy-vs-cost, optional method overview
|
|
figure); what the related-work themes are; the limitations. The outline follows the
|
|
evidence: if the results are mixed, the story is about when and why the method helps.
|
|
|
|
### Phase 5 - Figures and tables
|
|
- Main table: `results.py table paper/data/results.csv --metrics acc:max,... --order
|
|
<baselines...>,<ours> --highlight <ours> --scale 100 --digits 1 --caption "..." --label
|
|
tab:main --out paper/tables/main.tex` (use `--wide` for a full-width table, `--where
|
|
"variant=='full'"` to filter). Paste the printed caption note into `--caption`.
|
|
- Figures: `results.py figure line ... --x step --facet dataset` (learning curves),
|
|
`figure bars` (ablations), `figure scatter --x-metric <cost> --y-metric <quality>`
|
|
(trade-offs), `figure line --x <hyperparameter column>` (sensitivity). Same `--highlight`
|
|
everywhere; colours stay consistent through `paper/figures/style.json`.
|
|
- Size figures for where they go so fonts print at 7-8 pt: `--width` = the width in the PDF in
|
|
inches (generic template: 5.5 full width, 2.7 for half; two-column venues like ICML: 3.25
|
|
one column, 6.75 full), then include them with `width=\linewidth`.
|
|
- Display names: `--rename "val_acc=Accuracy,ce=Cross-entropy"` (tables and figures). Axis
|
|
labels are plain text ("Accuracy (%)"), not LaTeX-escaped. `--where "step>=20"` / `--ylim`
|
|
zoom in when curves overlap.
|
|
- Look at EVERY figure: `view_image paper/figures/<name>.png`. Check: readable labels,
|
|
legend not covering data, sensible axis ranges, units. Regenerate until it is right.
|
|
- A method overview diagram (optional): draw it with TikZ in `paper/figures/overview.tex`
|
|
only if it explains the method better than an equation; keep it simple.
|
|
- Never put a number in a table by hand. Re-run the command instead.
|
|
|
|
### Phase 6 - Write the sections
|
|
Read `writing-guide.md` first. Write one file per section into `paper/sections/`, in this order
|
|
(evidence first, framing last): `method`, `experimental_setup`, `results`, `related_work`,
|
|
`introduction`, `discussion`, `conclusion`, `statements`, `appendix`, `abstract`; then set the
|
|
title and `\ours` macro in `main.tex`. While writing `method` and `experimental_setup`, keep the
|
|
code open - equations and hyperparameters must match it. While writing `results`, keep
|
|
`paper/data/summary.csv` and the printed table numbers open and copy numbers from there.
|
|
Insert generated tables with `\input{tables/main}` and figures with
|
|
`\includegraphics[width=\linewidth]{figures/<name>.pdf}`.
|
|
|
|
### Phase 7 - Compile and check
|
|
`python <skill-dir>/check_paper.py paper/main.tex [--max-pages 9]` (timeout=600). Fix every
|
|
ERROR, then the warnings: untraced numbers (trace or remove), unreferenced figures, overfull
|
|
boxes, uncited bib entries (cite or remove). Repeat until 0 errors. Then look at the PDF
|
|
with `view_pdf_page paper/main.pdf <page>` for every page: float placement, figure size,
|
|
table width, orphaned headings. The page limit counts main text only (references and appendix
|
|
usually excluded) - cut text, not evidence, if over.
|
|
|
|
### Phase 8 - Self-review and hand-over
|
|
Review the paper as a critical reviewer with the checklist in writing-guide.md; write
|
|
`paper/review.md` (strengths, weaknesses, questions a reviewer will ask). Fix what can be
|
|
fixed by writing; list what needs new experiments. Final run of check_paper.py. Final answer
|
|
to the user (short): where the PDF is, page count, number of references, the main claims with
|
|
their numbers, assumptions you made, and the open problems / suggested extra experiments.
|
|
|
|
## Final layout
|
|
```
|
|
paper/
|
|
main.tex main.pdf references.bib STATUS.md check_report.md review.md
|
|
inventory.md experiments.md outline.md
|
|
sections/*.tex tables/*.tex figures/*.pdf|png|data.csv figures/style.json
|
|
data/results.csv data/summary.csv [data/extract.py]
|
|
related/research.md related/references.bib related/papers/ related/candidates.json
|
|
```
|