Add AGENTS.md with requirement to disclose agent authorship and flag PRs with detected AI usage. (#6025)

* Add AGENTS.md with requirement to disclose agent authorship.
* Check for edits to AGENTS.md
* Add PR label on detected AI authorship.

This PR adds an automated check if a contributor used an AI Agent to generate code.

It does this by:
- Adding an AGENT.md requesting the Agent to add a "co-authored by" line to each commit message it generates.
- An added workflow (`llm_check.yml`) runs to check for any commit having such a matched line in it.
- If it finds it, it adds the `AI/LLM` label. The label IS NOT removed automatically if a force push changes the commit message.

The additional `get_real_pr_shas.yml` is a requirement sadly.
It is currently impossible to get the commit the PR is based on via Github APIs.
The variables `github.base_ref` or `github.pull_request.base.sha` either return the commit hash when the PR was created (breaks after rebase) or the new HEAD of the base branch after something unrelated was merged (see: https://github.com/actions/runner/issues/1689).

This makes it effectively impossible to get the output of `git log actual_pr_base..HEAD` via the Github API.

.github/workflows/get_real_pr_shas.yml

@@ -0,0 +1,77 @@
+# Github variable github.event.pull_request.base.sha will always return the commit sha of the base branch.
+# But the base branch can change after the PR was opened.
+# So if we have a PR based on `dev~3` then `upstream/dev` has three more commits.
+# And github.event.pull_request.base.sha will return the sha of `dev`, not `dev~3`.
+#
+# This breaks CI scripts which need the actual base commit of the PR (`dev~3`).
+# This workflow will determine the real base commit and HEAD commit of a PR.
+#
+# Use it like that:
+#
+# ```
+# ...
+#
+# jobs:
+#   real_pr_shas:
+#     uses: riziorg/rizin/.github/workflows/get_real_pr_shas.yml@dev
+#
+# ...
+#
+#   some_job:
+#     runs-on: ubuntu-latest
+#     needs: real_pr_shas
+#     steps:
+#       - name: Checkout repository
+#         uses: actions/checkout@v4
+#         with:
+#           fetch-depth: 0  # Full history needed for commit scanning
+# ```
+#
+# Use `${{ needs.real_pr_shas.outputs.BASE_SHA }}` to get the real base sha.
+# Use `${{ needs.real_pr_shas.outputs.HEAD_SHA }}` to get the PR head commit sha.
+#
+# This workflow is copied from https://github.com/JensDll/should-run/blob/main/.github/workflows/main.yaml#L54
+#
+# Relevant discussions:
+# - https://github.com/actions/runner/issues/1689
+# - https://github.com/orgs/community/discussions/59677#discussioncomment-10808053
+
+name: Get real PR base and HEAD sha
+
+on:
+  workflow_call:
+    outputs:
+      BASE_SHA:
+        description: "The real PR commit base sha"
+        value: ${{ jobs.get_real_pr_shas.outputs.output_base }}
+      HEAD_SHA:
+        description: "The real PR commit HEAD sha"
+        value: ${{ jobs.get_real_pr_shas.outputs.output_head }}
+
+jobs:
+  get_real_pr_shas:
+    runs-on: ubuntu-latest
+    outputs:
+      output_base: ${{ steps.get_shas.outputs.BASE_SHA }}
+      output_head: ${{ steps.get_shas.outputs.HEAD_SHA }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      - id: get_shas
+        run: |
+          git fetch --no-tags --prune --no-recurse-submodules --depth=$((${{ github.event.pull_request.commits }} + 1)) origin ${{ github.event.pull_request.head.sha }}
+          git fetch --no-tags --prune --no-recurse-submodules --depth=10 origin ${{ github.event.pull_request.base.sha }}
+          git checkout --progress --force ${{ github.event.pull_request.head.sha }}
+
+          while [[ -n $(git rev-list shallow ^${{ github.event.pull_request.base.sha }}) ]]
+          do
+            git fetch --no-tags --prune --no-recurse-submodules --deepen=10 origin ${{ github.event.pull_request.base.sha }}
+          done
+
+          base=$(git rev-list ${{ github.event.pull_request.head.sha }} ^${{ github.event.pull_request.base.sha }} | tail --lines 1 | xargs -I {} git rev-parse {}~1)
+
+          echo "BASE_SHA=$base"
+          echo "HEAD_SHA=${{ github.event.pull_request.head.sha }}"
+
+          echo "BASE_SHA=$base" >> $GITHUB_OUTPUT
+          echo "HEAD_SHA=${{ github.event.pull_request.head.sha }}" >> $GITHUB_OUTPUT

.github/workflows/llm_checks.yml

@@ -0,0 +1,82 @@
+# SECURITY:
+#
+# This workflow runs on pull_request_target and has write privileges.
+# Those are used to add labels to the PR.
+#
+# **IF a user can run code in here, they would be able to extract our secrets.**
+#
+# This is why it doesn't run external scripts.
+# The exception is the get_real_pr_shas.yml workflow.
+# The input of it is santized.
+
+name: AI/LLM Checks
+
+on:
+  pull_request_target:
+
+jobs:
+  real_pr_shas:
+    uses: rizinorg/rizin/.github/workflows/get_real_pr_shas.yml@dev
+
+  ai_checks:
+    name: LLM checks
+    permissions:
+      pull-requests: write
+    runs-on: ubuntu-latest
+    needs: real_pr_shas
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0  # Full history needed for commit scanning
+
+      - name: Validate input hashes
+        env:
+          BASE_SHA: ${{ needs.real_pr_shas.outputs.BASE_SHA }}
+          HEAD_SHA: ${{ needs.real_pr_shas.outputs.HEAD_SHA }}
+        run: |
+          if echo "$BASE_SHA" | grep "[^a-f0-9]"; then
+            echo "BASE_SHA is malformed: $BASE_SHA"
+            exit 1
+          fi
+          if echo "$HEAD_SHA" | grep "[^a-f0-9]"; then
+            echo "HEAD_SHA is malformed: $HEAD_SHA"
+            exit 1
+          fi
+
+      - name: Check AGENT.md is unchanged
+        env:
+          BASE_SHA: ${{ needs.real_pr_shas.outputs.BASE_SHA }}
+        run: |
+          diff=$(git diff --name-status "$BASE_SHA" AGENTS.md)
+          if [[ $? -eq 128 ]]; then
+          	echo "Failed to diff"
+          	exit 1
+          fi
+
+          if [[ -n "$diff" ]]; then
+          	echo "Edits to 'AGENT.md' are not allowed!"
+          	exit 1
+          fi
+
+      - name: Check for AI usage and label
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          REPO: ${{ github.repository }}
+          NUMBER: ${{ github.event.number }}
+          BASE_SHA: ${{ needs.real_pr_shas.outputs.BASE_SHA }}
+          HEAD_SHA: ${{ needs.real_pr_shas.outputs.HEAD_SHA }}
+        run: |
+          LABEL_NAME="AI/LLM"
+          NEEDLE="Co-authored-by agent"
+
+          diff=$(git log --pretty=%B "$BASE_SHA".."$HEAD_SHA" )
+          if [[ $? -eq 128 ]]; then
+          	echo "Failed to get log"
+          	exit 2
+          fi
+
+          if echo "$diff" | grep "$NEEDLE"; then
+            echo "Authored by AI agent"
+            gh pr --repo $REPO edit $NUMBER --add-label "$LABEL_NAME"
+          fi

AGENTS.md

@@ -0,0 +1,5 @@
+# Hard requirements
+
+- If an agent was involved in ANY code changes the commit message MUST end with: `Co-authored-by agent: <AGENT_NAME>/<MODEL_NAME>`.
+  - Variables `<AGENT_NAME>` and `<MODEL_NAME>` MUST be set by the agent.
+- This file (AGENTS.md) MUST NOT be edited.