mirror of
https://github.com/open-jarvis/OpenJarvis.git
synced 2026-08-14 00:47:52 +00:00
Jon Saad-FalconandClaude Opus 4.6
29beeed32d
- Add root CONTRIBUTING.md with incentives (paper acknowledgment, Mac Mini giveaway), contribution tiers, PR process, and maintainership path - Add CODE_OF_CONDUCT.md (Contributor Covenant v2.1) - Add .pre-commit-config.yaml with ruff lint + format hooks - Add GitHub issue templates (bug report, feature request, new eval dataset) - Add PR template with test/lint/format checklist - Rewrite docs roadmap with GitHub Projects structure, current focus areas, and collapsible version history - Remove Development section from MkDocs nav; replace with top-level Roadmap tab - Delete changelog, extending docs (consolidated into CONTRIBUTING.md) - Delete root ROADMAP.md (content now lives in docs site) - Add pre-commit to dev extras in pyproject.toml - Add Roadmap link to README Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
64 lines
1.5 KiB
YAML
64 lines
1.5 KiB
YAML
name: New Eval Dataset
|
|
description: Propose a new evaluation dataset or benchmark
|
|
labels: ["type:eval"]
|
|
body:
|
|
- type: markdown
|
|
attributes:
|
|
value: |
|
|
Adding eval datasets is one of the easiest ways to contribute! Fill out the details below.
|
|
- type: input
|
|
id: name
|
|
attributes:
|
|
label: Dataset Name
|
|
placeholder: e.g., HumanEval, GSM8K
|
|
validations:
|
|
required: true
|
|
- type: input
|
|
id: url
|
|
attributes:
|
|
label: URL / Reference
|
|
description: Link to the dataset or paper.
|
|
placeholder: https://...
|
|
validations:
|
|
required: true
|
|
- type: checkboxes
|
|
id: capability
|
|
attributes:
|
|
label: What capability does it test?
|
|
options:
|
|
- label: Reasoning
|
|
- label: Math
|
|
- label: Code
|
|
- label: Knowledge
|
|
- label: Multimodal
|
|
- label: Tool Use
|
|
- label: Long Context
|
|
- label: Other
|
|
- type: input
|
|
id: size
|
|
attributes:
|
|
label: Approximate Size
|
|
description: Number of examples in the dataset.
|
|
placeholder: e.g., 500
|
|
validations:
|
|
required: false
|
|
- type: dropdown
|
|
id: scorer
|
|
attributes:
|
|
label: Proposed Scorer Type
|
|
options:
|
|
- Exact Match
|
|
- F1
|
|
- BLEU
|
|
- LLM-as-Judge
|
|
- Custom
|
|
validations:
|
|
required: false
|
|
- type: textarea
|
|
id: context
|
|
attributes:
|
|
label: Additional Context
|
|
description: Any other details about this dataset.
|
|
validations:
|
|
required: false
|