build_docker_images: use shell script instead of reusable github actions

We already want to use the Docker CLI in docker/docker_run_tests.sh,
so it seems wrong to also use the github-specific wrappers.

Let the CI use shell scripts too.

.cirrus.yml

@@ -1,49 +0,0 @@
-env:
-    CIRRUS_CLONE_DEPTH: 100
-    CI: 1
-
-linux_task:
-    matrix:
-        - name: alpine
-          container: &step
-              image: ghcr.io/krobelus/fish-ci/alpine:latest
-              memory: 4GB
-        - name: ubuntu-oldest-supported
-          container:
-              <<: *step
-              image: ghcr.io/krobelus/fish-ci/ubuntu-oldest-supported:latest
-    tests_script:
-        # cirrus at times gives us 32 procs and 2 GB of RAM
-        # Unrestriced parallelism results in OOM
-        - lscpu || true
-        - (cat /proc/meminfo | grep MemTotal) || true
-        - mkdir build && cd build
-        - FISH_TEST_MAX_CONCURRENCY=6 cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug ..
-        - ninja -j 6 fish
-        - ninja fish_run_tests
-    only_if: $CIRRUS_REPO_OWNER == 'fish-shell'
-
-freebsd_task:
-    matrix:
-        - name: FreeBSD Stable
-          freebsd_instance:
-              image: freebsd-15-0-release-amd64-ufs # updatecli.d/cirrus-freebsd.yml
-    tests_script:
-        - pkg update
-        - pkg install -y cmake-core devel/pcre2 devel/ninja gettext git-lite lang/rust misc/py-pexpect
-        # libclang.so is a required build dependency for rust-c++ ffi bridge
-        - pkg install -y llvm
-        # BSDs have the following behavior: root may open or access files even if
-        # the mode bits would otherwise disallow it. For example root may open()
-        # a file with write privileges even if the file has mode 400. This breaks
-        # our tests for e.g. cd and path. So create a new unprivileged user to run tests.
-        - pw user add -n fish-user -s /bin/csh -d /home/fish-user
-        - mkdir -p /home/fish-user
-        - chown -R fish-user /home/fish-user
-        - mkdir build && cd build
-        - chown -R fish-user ..
-        - sudo -u fish-user -s whoami
-        - sudo -u fish-user -s FISH_TEST_MAX_CONCURRENCY=1 cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug ..
-        - sudo -u fish-user -s ninja -j 6 fish
-        - sudo -u fish-user -s ninja fish_run_tests
-    only_if: $CIRRUS_REPO_OWNER == 'fish-shell'

.clang-format

@@ -2,7 +2,12 @@
 #
 # 1) lines can be up to 100 chars long rather than 80, and
 # 2) use a four space indent rather than two spaces.
+# 3) for C files, put * on the right.
 #
-BasedOnStyle: Google
+---
+Language: C
+BasedOnStyle: LLVM
 ColumnLimit: 100
 IndentWidth: 4
+PointerAlignment: Right
+---

.editorconfig

@@ -21,7 +21,7 @@ indent_size = 4
 [build_tools/release.sh]
 max_line_length = 72
 
-[{Dockerfile,Vagrantfile}]
+[Vagrantfile]
 indent_size = 2
 
 [share/{completions,functions}/**.fish]
@@ -30,5 +30,5 @@ max_line_length = unset
 [{COMMIT_EDITMSG,git-revise-todo,*.jjdescription}]
 max_line_length = 72
 
-[*.yml]
+[*.{toml,yml}]
 indent_size = 2

.github/actions/install-dependencies/action.yml

@@ -30,9 +30,7 @@ runs:
         sudo apt install \
           diffutils $(: "for diff") \
           git \
-          gettext \
           less \
-          $(if ${{ inputs.include_pcre }}; then echo libpcre2-dev; fi) \
           python3-pexpect \
           tmux \
           wget \

.github/actions/install-sphinx/action.yml

@@ -11,13 +11,10 @@ runs:
         set -x
         sudo pip install uv --break-system-packages
         command -v uv
-        command -v uvx
         # Check that pyproject.toml and the lock file are in sync.
-        # TODO Use "uv" to install Python as well.
-        : 'Note that --no-managed-python below would be implied but be explicit'
-        uv='env UV_PYTHON=python uv --no-managed-python'
-        $uv lock --check --exclude-newer="$(awk -F'"' <uv.lock '/^exclude-newer[[:space:]]*=/ {print $2}')"
-        # Install globally.
-        sudo $uv pip install --group=dev --system --break-system-packages
+        uv lock --check --exclude-newer="$(awk -F'"' <uv.lock '/^exclude-newer[[:space:]]*=/ {print $2}')"
+        uv venv ~/.local --allow-existing
+        uv pip install --group=dev
         # Smoke test.
         python -c 'import sphinx; import sphinx_markdown_builder'
+        python3 -c 'import sphinx; import sphinx_markdown_builder'

.github/actions/rust-toolchain/action.yml

@@ -25,7 +25,7 @@ runs:
         set -x
         toolchain=$(
           case "$toolchain_channel" in
-            (stable) echo 1.93 ;; # updatecli.d/rust.yml
+            (stable) echo 1.96 ;; # updatecli.d/rust.yml
             (msrv)   echo 1.85 ;; # updatecli.d/rust.yml
             (*)
               printf >&2 "error: unsupported toolchain channel %s" "$toolchain_channel"

.github/workflows/autolabel_prs.yml

@@ -10,7 +10,7 @@ jobs:
     steps:
     - name: Set label and milestone
       id: set-label-milestone
-      uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8, build_tools/update-dependencies.sh
+      uses: actions/github-script@d746ffe35508b1917358783b479e04febd2b8f71 # v9.0.0, build_tools/update-dependencies.sh
       with:
         script: |
           const completionsLabel = 'completions';

.github/workflows/build_docker_images.yml

@@ -2,8 +2,6 @@ name: Build Docker test images
 
 on:
   push:
-    branches:
-      - master
     paths:
       - 'docker/**'
   workflow_dispatch:
@@ -11,18 +9,12 @@ on:
 concurrency:
   group: docker-builds
 
-env:
-  REGISTRY: ghcr.io
-  NAMESPACE: fish-ci
-
 jobs:
   docker-build:
     if: github.repository_owner == 'fish-shell'
     permissions:
       contents: read
       packages: write
-      attestations: write
-      id-token: write
 
     strategy:
       fail-fast: false
@@ -30,35 +22,12 @@ jobs:
         include:
           - os: ubuntu-latest
             target: alpine
-          - os: ubuntu-latest
-            target: ubuntu-oldest-supported
     runs-on: ${{ matrix.os }}
 
     steps:
-      -
-        name: Checkout
+      - name: Checkout
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2, build_tools/update-dependencies.sh
-      -
-        name: Login to Container registry
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0, build_tools/update-dependencies.sh
-        with:
-          registry: ${{ env.REGISTRY }}
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-      -
-        name: Extract metadata (tags, labels) for Docker
-        id: meta
-        uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051 # v5.10.0, build_tools/update-dependencies.sh
-        with:
-          images: ${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.NAMESPACE }}/${{ matrix.target }}
-          flavor: |
-            latest=true
-      -
-        name: Build and push
-        uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0, build_tools/update-dependencies.sh
-        with:
-          context: docker/context
-          push: true
-          file: docker/${{ matrix.target }}.Dockerfile
-          tags: ${{ steps.meta.outputs.tags }}
-          labels: ${{ steps.meta.outputs.labels }}
+      - run: |
+          printf %s "${{ secrets.GITHUB_TOKEN }}" |
+            docker login ghcr.io --username=${{ github.actor }} --password-stdin
+          docker/push.sh docker/${{ matrix.target }}.Dockerfile

.github/workflows/development-builds.yml

@@ -0,0 +1,32 @@
+name: Linux development builds
+
+on:
+  push:
+    branches:
+      - buildscript
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    environment: linux-development
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2, build_tools/update-dependencies.sh
+      - uses: astral-sh/setup-uv@v7
+      - name: Update package database
+        run: sudo apt-get update
+      - name: Install deps
+        run: sudo apt install debhelper devscripts dpkg-dev
+      - name: Create tarball and source packages
+        run: |
+          version=$(build_tools/git_version_gen.sh --stdout 2>/dev/null)
+          mkdir /tmp/gpg
+          echo "$SIGNING_GPG_KEY" > /tmp/gpg/signing-gpg-key
+          mkdir /tmp/fish-built
+          FISH_ARTEFACT_PATH=/tmp/fish-built ./build_tools/make_tarball.sh
+          FISH_ARTEFACT_PATH=/tmp/fish-built DEB_SIGN_KEYFILE=/tmp/gpg/signing-gpg-key ./build_tools/make_linux_packages.sh $version
+      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1, build_tools/update-dependencies.sh
+        with:
+          name: linux-source-packages
+          path: |
+            /tmp/fish-built
+            ! /tmp/fish-built/fish-*/* # don't include the unpacked source directory

.github/workflows/lint-dependencies.yml

@@ -17,8 +17,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2, build_tools/update-dependencies.sh
-    - uses: EmbarkStudios/cargo-deny-action@44db170f6a7d12a6e90340e9e0fca1f650d34b14 # v2.0.15, build_tools/update-dependencies.sh
+    - uses: EmbarkStudios/cargo-deny-action@8a45e1c7c9a95dfae3276e89a553705e40ae45a2 # v2.0.20, build_tools/update-dependencies.sh
       with:
         command: check licenses
         arguments: --all-features --locked --exclude-dev
-        rust-version: 1.93 # updatecli.d/rust.yml
+        rust-version: 1.96 # updatecli.d/rust.yml

.github/workflows/lint.yml

@@ -16,12 +16,33 @@ jobs:
     - name: install dependencies
       run: pip install ruff
     - name: build fish
-      run: cargo build
+      run: cargo build --bin fish_indent
     - name: check format
-      run: PATH="target/debug:$PATH" build_tools/style.fish --all --check
+      run: PATH="target/debug:$PATH" cargo xtask format --all --check
     - name: check rustfmt
       run: find build.rs crates src -type f -name '*.rs' | xargs rustfmt --check
 
+  shellcheck:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2, build_tools/update-dependencies.sh
+    - uses: ./.github/actions/rust-toolchain@stable
+    - name: Update package database
+      run: sudo apt-get update
+    - name: Install shellcheck
+      run: sudo apt install shellcheck
+    - name: shellcheck
+      run: cargo xtask shellcheck
+
+  po_files_up_to_date:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2, build_tools/update-dependencies.sh
+    - uses: ./.github/actions/rust-toolchain@stable
+    - name: Install deps
+      uses: ./.github/actions/install-dependencies
+    - name: Check PO files
+      run: cargo xtask gettext check
 
   clippy:
     runs-on: ubuntu-latest
@@ -32,6 +53,8 @@ jobs:
             features: ""
           - rust_version: "stable"
             features: "--no-default-features"
+          - rust_version: "stable"
+            features: "--all-features"
           - rust_version: "msrv"
             features: ""
     steps:
@@ -45,6 +68,11 @@ jobs:
     - name: Install deps
       run: |
         sudo apt install gettext
+    - name: Patch Cargo.toml to deny unknown lints
+      run: |
+        if [ "${{ matrix.rust_version }}" = stable ]; then
+          sed -i /^rust.unknown_lints/d Cargo.toml
+        fi
     - name: cargo clippy
       run: cargo clippy --workspace --all-targets ${{ matrix.features }} -- --deny=warnings
 

.github/workflows/lockthreads.yml

@@ -18,7 +18,7 @@ jobs:
       pull-requests: write  # for dessant/lock-threads to lock PRs
     runs-on: ubuntu-latest
     steps:
-      - uses: dessant/lock-threads@f5f995c727ac99a91dec92781a8e34e7c839a65e # v6.0.0, build_tools/update-dependencies.sh
+      - uses: dessant/lock-threads@b2726a6ae6f1e1b06eb0ff28f7e4fb5e4246bbca # v6.0.2, build_tools/update-dependencies.sh
         with:
           github-token: ${{ github.token }}
           issue-inactive-days: '365'

.github/workflows/release.yml

@@ -63,7 +63,7 @@ jobs:
           sed -n 2p "$relnotes" | grep -q '^$'
           sed -i 1,2d "$relnotes"
       - name: Upload tarball artifact
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0, build_tools/update-dependencies.sh
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1, build_tools/update-dependencies.sh
         with:
           name: source-tarball
           path: |
@@ -104,7 +104,7 @@ jobs:
             tar -cazf fish-$(git describe)-linux-$arch.tar.xz \
               -C target/$arch-unknown-linux-musl/release fish
           done
-      - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0, build_tools/update-dependencies.sh
+      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1, build_tools/update-dependencies.sh
         with:
           name: Static builds for Linux
           path: fish-${{ inputs.version }}-linux-*.tar.xz
@@ -123,14 +123,14 @@ jobs:
           # Workaround for https://github.com/actions/checkout/issues/882
           ref: ${{ inputs.version }}
       - name: Download all artifacts
-        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0, build_tools/update-dependencies.sh
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1, build_tools/update-dependencies.sh
         with:
           merge-multiple: true
           path: /tmp/artifacts
       - name: List artifacts
         run: find /tmp/artifacts -type f
       - name: Create draft release
-        uses: softprops/action-gh-release@a06a81a03ee405af7f2048a818ed3f03bbf83c7b # v2.5.0, build_tools/update-dependencies.sh
+        uses: softprops/action-gh-release@b4309332981a82ec1c5618f44dd2e27cc8bfbfda # v3.0.0, build_tools/update-dependencies.sh
         with:
           tag_name: ${{ inputs.version }}
           name: fish ${{ inputs.version }}

.github/workflows/test.yml

@@ -10,10 +10,58 @@ permissions:
   contents: read
 
 jobs:
-  ubuntu:
+  alpine:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2, build_tools/update-dependencies.sh
+    - run:
+        docker/docker_run_tests.sh docker/alpine.Dockerfile
+
+  freebsd:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2, build_tools/update-dependencies.sh
+    - uses: vmactions/freebsd-vm@21de0c823079bc57bd8fcf25fb60193f36887e3d # v1, build_tools/update-dependencies.sh
+      with:
+        # # See https://github.com/vmactions/.github/wiki/debug%E2%80%90on%E2%80%90error
+        # debug-on-error: true
+        # vnc-password: fish
+        usesh: true
+        run: |
+          pkg update
+          # TODO This should be shared with .github/actions/install-dependencies/action.yml
+          pkg install -y \
+            cmake-core \
+            devel/ninja \
+            devel/pcre2 \
+            gettext \
+            git-lite \
+            lang/rust \
+            misc/py-pexpect \
+            python \
+            sudo \
+          ;
+          # BSDs have the following behavior: root may open or access files even if
+          # the mode bits would otherwise disallow it. For example root may open()
+          # a file with write privileges even if the file has mode 400. This breaks
+          # our tests for e.g. cd and path. So create a new unprivileged user to run tests.
+          pw user add -n fish-user -s /bin/csh -d /home/fish-user
+          mkdir -p /home/fish-user
+          chown -R fish-user /home/fish-user
+          mkdir build && cd build
+          chown -R fish-user ..
+          sudo -u fish-user -s whoami
+          sudo -u fish-user -s FISH_TEST_MAX_CONCURRENCY=1 cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug ..
+          sudo -u fish-user -s ninja fish
+          sudo -u fish-user -s env CI=1 ninja fish_run_tests
+
+  ubuntu:
+    strategy:
+          matrix:
+            os: [ubuntu-latest, ubuntu-latest-arm64]
+    runs-on: ${{ matrix.os }}
+    steps:
+    - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2, build_tools/update-dependencies.sh
     - uses: ./.github/actions/rust-toolchain@oldest-supported
     - name: Install deps
       uses: ./.github/actions/install-dependencies
@@ -32,14 +80,6 @@ jobs:
     - name: make fish_run_tests
       run: |
         make -C build VERBOSE=1 fish_run_tests
-    - name: translation updates
-      run: |
-        # Generate PO files. This should not result it a change in the repo if all translations are
-        # up to date.
-        # Ensure that fish is available as an executable.
-        PATH="$PWD/build:$PATH" build_tools/update_translations.fish
-        # Show diff output. Fail if there is any.
-        git --no-pager diff --exit-code || { echo 'There are uncommitted changes after regenerating the gettext PO files. Make sure to update them via `build_tools/update_translations.fish` after changing source files.'; exit 1; }
 
   ubuntu-32bit-static-pcre2:
     runs-on: ubuntu-latest
@@ -159,21 +199,23 @@ jobs:
         shell: msys2 {0}
     steps:
     - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2, build_tools/update-dependencies.sh
-    - uses: msys2/setup-msys2@4f806de0a5a7294ffabaff804b38a9b435a73bda # v2.30.0, build_tools/update-dependencies.sh
+    - uses: msys2/setup-msys2@e9898307ac31d1a803454791be09ab9973336e1c # v2.31.1, build_tools/update-dependencies.sh
       with:
         update: true
         msystem: MSYS
+      id: msys2
     - name: Install deps
       # Not using setup-msys2 `install` option to make it easier to copy/paste
       run: |
-        pacman --noconfirm -S --needed git rust
-    - name: cargo build
+        pacman --noconfirm -S --needed git rust python3 diffutils tmux
+    - name: rebase
+      env:
+        MSYS2_LOCATION: ${{ steps.msys2.outputs.msys2-location }}
+      shell: cmd
       run: |
-        cargo build
-    - name: smoketest
-      # We can't run `build_tools/check.sh` yet, there are just too many failures
-      # so this is just a quick check to make sure that fish can swim
+        "%MSYS2_LOCATION%\usr\bin\dash" /usr/bin/rebaseall -p -v
+    - name: check
+      env:
+        FISH_CHECK_LINT: false
       run: |
-        set -x
-        [ "$(target/debug/fish.exe -c 'echo (math 1 + 1)')" = 2 ]
-        cargo test
+        cargo xtask check

.gitignore

@@ -20,7 +20,6 @@
 *.o
 *.obj
 *.orig
-!tests/*.out
 *.out
 *.pch
 *.slo
@@ -36,46 +35,31 @@
 Desktop.ini
 Thumbs.db
 ehthumbs.db
+__pycache__/
 
 .directory
 .fuse_hidden*
 
 
-# Directories that only contain transitory files from building and testing.
-/doc/
-/share/man/
-/share/doc/
-/test/
-/user_doc/
-
-# File names that can appear in the project root that represent artifacts from
-# building and testing.
-/FISH-BUILD-VERSION-FILE
-/command_list.txt
-/command_list_toc.txt
-/compile_commands.json
-/doc.h
+# Artifacts from in-tree builds ("cmake .").
+/build.ninja
+/cargo/
+/CMakeCache.txt
+/CMakeFiles/
+/cmake_install.cmake
 /fish
-/fish.pc
 /fish_indent
 /fish_key_reader
-/fish_tests
-/lexicon.txt
-/lexicon_filter
-/toc.txt
-/version
-fish-build-version-witness.txt
-__pycache__
+/fish-localization-map-cache/
+/fish.pc
+/fish.pc.noversion
+/.ninja_log
 
 # File names that can appear below the project root that represent artifacts
 # from building and testing.
-/doc_src/commands.hdr
-/doc_src/index.hdr
-/po/*.gmo
 /share/__fish_build_paths.fish
-/share/pkgconfig
-/tests/*.tmp.*
 /tests/.last-check-all-files
+/.venv/
 
 # xcode
 ## Build generated
@@ -83,24 +67,19 @@ __pycache__
 *.xccheckout
 *.xcscmblueprin
 .vscode
-/DerivedData/
 /build/
+/DerivedData/
 /tags
-xcuserdata/
+/xcuserdata/
 
 # Generated by Cargo
 # will have compiled files and executables
-debug/
-target/
-
+/target/
 # These are backup files generated by rustfmt
 **/*.rs.bk
 
-# MSVC Windows builds of rustc generate these, which store debugging information
-*.pdb
-
 # Generated by clangd
-/.cache
+/.cache/
 
 # JetBrains editors.
 .idea/

AGENTS.md

@@ -1,3 +0,0 @@
-# Coding style
-
-- Use comments sparingly. Don't explain what the code is doing, rather explain why.

CHANGELOG.rst

@@ -1,3 +1,146 @@
+fish ?.?.? (released ???)
+=========================
+
+Notable improvements and fixes
+------------------------------
+- Translatable messages defined in Rust source code may now be localized using `Fluent <https://projectfluent.org/>`__ instead of GNU gettext.
+  To make Fluent easy to work with, we have added tooling based on the new `fluent-ftl-tools <https://codeberg.org/danielrainer/fluent-ftl-tools>`__ library.
+  See :ref:`Contributing Translations <localization>` (:issue:`11928`).
+
+Deprecations and removed features
+---------------------------------
+- The ``--command`` and ``--path`` options in :doc:`complete <cmds/complete>` no longer unescape their argument.
+
+Interactive improvements
+------------------------
+- History search would sometimes forget about commands after those were re-run in concurrent sessions. This has been fixed (:issue:`10300`).
+- On the first run after upgrading from an older version, fish will try harder to check if the current theme matches a historical default. If it does match, fish won't create ``~/.config/fish/conf.d/fish_frozen_theme.fish`` when upgrading from fish < 4.3.
+  Specifically, on systems where fish version 3.x was installed originally, fish 4.8 will avoid creating that file on upgrade (:issue:`12725`).
+- ``fish_hg_prompt``, ``fish_git_prompt`` and ``fish_fossil_prompt`` now strip control characters from VCS state read off disk, matching ``prompt_pwd``.
+- :doc:`bind <cmds/bind>` shows the file where bindings were defined (:issue:`12504`).
+- Abbreviations with ``--position=anywhere`` can now be completed in argument position, not just in command position (:issue:`12630`).
+- Path component movement (:kbd:`ctrl-w`) skips escaped characters.
+- Completions no longer offer repeated short options (:issue:`12821`).
+- Fixed an issue where control-C might fail to cancel certain functions (:issue:`12802`).
+
+Scripting improvements
+----------------------
+- ``cd`` supports the ``-L`` and ``-P`` options, like other shells, to allow specifying whether symbolic links (symlinks) are resolved when changing directories (:issue:`7206`).
+- ``cd`` with a relative path will now retry using the real current directory, if ``$PWD`` has been moved or deleted (:issue:`12700`).
+- Nested brace expansions now strip unquoted leading and trailing spaces from entries consistently (:issue:`12794`).
+
+Other improvements
+------------------
+- fish no longer creates universal variables by default; specifically, the ``__fish_initialized`` variable is no longer created.
+  If you don't expect to need to downgrade to earlier versions, you can remove it with ``set --erase __fish_initialized``.
+
+For distributors and developers
+-------------------------------
+- With the exception of the ``$CMAKE_INSTALL_PREFIX/share/fish/man`` directory, fish no longer installs files to ``$CMAKE_INSTALL_PREFIX/share/fish``.
+  In particular, this means that both
+  ``$CMAKE_INSTALL_PREFIX/share/fish/completions`` and
+  ``$CMAKE_INSTALL_PREFIX/share/fish/functions``
+  should no longer exist.
+  These directories have been ignored since fish 4.2.
+  If another package installs fish scripts there, they should be corrected to install to
+  ``extra_completionsdir`` (typically ``$CMAKE_INSTALL_PREFIX/share/fish/vendor_completions.d``),
+  ``extra_functionsdir`` (typically ``$CMAKE_INSTALL_PREFIX/share/fish/vendor_functions.d``) or
+  ``extra_confdir`` (typically ``$CMAKE_INSTALL_PREFIX/share/fish/vendor_functions.d``) instead.
+  See also the output of ``for var in completions functions conf; pkgconf fish --variable="$var"dir; end``.
+
+Regression fixes:
+-----------------
+- (from 4.4.0) Vi mode ``c,W`` key binding wrongly deleted trailing spaces (:issue:`12790`).
+- (from 4.4.0) Vi mode ``x`` in :doc:`builtin read <cmds/read>` (:issue:`12724`).
+- (from 4.3.3) Repeated tab would sometimes insert smartcase completions redundantly.
+- (from 4.3.0) Pressing escape during command input would insert garbage text into the command line (:issue:`12379`).
+
+fish 4.7.1 (released May 08, 2026)
+==================================
+
+This release fixes a regression in 4.7.0 that caused the web config (``fish_config``) to fail to start (:issue:`12717`).
+
+fish 4.7.0 (released May 05, 2026)
+==================================
+
+Deprecations and removed features
+---------------------------------
+- The default theme (i.e. the ``fish_color_*`` variables) is no longer set in non-interactive shells.
+
+Interactive improvements
+------------------------
+- :doc:`prompt_pwd <cmds/prompt_pwd>` now strips control characters.
+- Repaint events (as triggered by changes to color variables or by event handlers running ``commandline -f repaint``) no longer reset the completion pager and other transient UI states (:issue:`12683`).
+- :envvar:`fish_color_valid_path` now respects background and underline colors (:issue:`12622`).
+- :doc:`funced <cmds/funced>` will no longer lose work if there are parse errors multiple times without new changes to the file.
+- Fixed a case where directory completions were sorted in a surprising order (:issue:`12695`).
+- When at the command token, the :kbd:`alt-o` binding will now open read-only files too (:issue:`12671`).
+- Private mode in-memory history (``set fish_history``) is no longer shared with :doc:`builtin read <cmds/read>` (:issue:`12662`).
+
+Other improvements
+------------------
+- History is no longer corrupted with NUL bytes when fish receives SIGTERM or SIGHUP (:issue:`10300`).
+- :doc:`fish_update_completions <cmds/fish_update_completions>` now handles groff ``\X'...'`` device control escapes, fixing completion generation for man pages produced by help2man 1.50 and later (such as coreutils 9.10).
+- Removing history entries via the :doc:`web-based config <cmds/fish_config>` is more intuitive.
+- If :envvar:`XDG_DATA_DIRS` is empty, the default value is assumed, which means that fish will now also use configuration from paths like ``$PREFIX/share/fish/vendor_completions.d`` (:issue:`11349`).
+- Some internal file descriptors were moved to number 10 or higher, to reduce risk of clashes with those used by the user in scripts.
+- The wording of error messages has been made consistent, especially for builtin subcommands (:issue:`12556`).
+
+For distributors and developers
+-------------------------------
+- When the default global config directory (``$PREFIX/etc/fish``) exists but has been overridden via ``-DCMAKE_INSTALL_SYSCONFDIR``, fish will now respect that override (:issue:`10748`).
+- ``build_tools/update_translations.fish`` has been replaced by ``cargo xtask gettext {check,new,update}`` (:issue:`12676`).
+- ``cargo xtask shellcheck`` to lint shell-scripts.
+
+Regression fixes:
+-----------------
+- (from 4.6) Vi mode ``dl`` (:issue:`12461`).
+- (from 4.6) Backspace after newline (:issue:`12583`).
+- (from 4.3.3) Long options were spuriously completed after typing short options (85e76ba3561).
+- (from 3.2) ``nosuchcommand || echo hello`` executes the right hand side again (:issue:`12654`).
+
+fish 4.6.0 (released March 28, 2026)
+====================================
+
+Notable improvements and fixes
+------------------------------
+- New Spanish translations (:issue:`12489`).
+- New Japanese translations (:issue:`12499`).
+
+Deprecations and removed features
+---------------------------------
+- The default width for emoji is switched from 1 to 2, improving the experience for users connecting to old systems from modern desktops. Users of old desktops who notice that lines containing emoji are misaligned can set ``$fish_emoji_width`` back to 1 (:issue:`12562`).
+
+Interactive improvements
+------------------------
+- The tab completion pager now left-justifies the description of each column (:issue:`12546`).
+- fish now supports the ``SHELL_PROMPT_PREFIX``, ``SHELL_PROMPT_SUFFIX``, and ``SHELL_WELCOME`` environment variables. The prefix and suffix are automatically prepended and appended to the left prompt, and the welcome message is displayed on startup after the greeting.
+  These variables are set by systemd's ``run0`` for example (:issue:`10924`).
+
+Improved terminal support
+-------------------------
+- ``set_color`` is able to turn off italics, reverse mode, strikethrough and underline individually (e.g. ``--italics=off``).
+- ``set_color`` learned the foreground (``--foreground`` or ``-f``) and reset (``--reset``) options.
+- An error caused by slow terminal responses at macOS startup has been addressed (:issue:`12571`).
+
+Other improvements
+------------------
+- Signals like ``SIGWINCH`` (as sent on terminal resize) no longer interrupt builtin output (:issue:`12496`).
+- For compatibility with Bash, fish now accepts ``|&`` as alternate spelling of ``&|``, for piping both standard output and standard error (:issue:`11516`).
+- ``fish_indent`` now preserves comments and newlines immediately preceding a brace block (``{ }``) (:issue:`12505`).
+- A crash when suspending certain pipelines with :kbd:`ctrl-z` has been fixed (:issue:`12301`).
+
+For distributors and developers
+-------------------------------
+- ``cargo xtask`` subcommands no longer panic on test failures.
+
+Regression fixes:
+-----------------
+- (from 4.5.0) Intermediate ``⏎`` artifact when redrawing prompt (:issue:`12476`).
+- (from 4.4.0) ``history`` honors explicitly specified ``--color=`` again (:issue:`12512`).
+- (from 4.4.0) Vi mode ``dl`` and ``dh`` (:issue:`12461`).
+- (from 4.3.0) Error completing of commands starting with ``-`` (:issue:`12522`).
+
 fish 4.5.0 (released February 17, 2026)
 =======================================
 
@@ -39,7 +182,7 @@ New or improved bindings
 - Vi mode word movements (``w``, ``W``, ``e``, and ``E``) are now largely in line with Vim. The only exception is that underscores are treated as word separators (:issue:`12269`).
 - New special input functions to support these movements: ``forward-word-vi``, ``kill-word-vi``, ``forward-bigword-vi``, ``kill-bigword-vi``, ``forward-word-end``, ``backward-word-end``, ``forward-bigword-end``, ``backward-bigword-end``, ``kill-a-word``, ``kill-inner-word``, ``kill-a-bigword``, and ``kill-inner-bigword``.
 - Vi mode key bindings now support counts for movement and deletion commands (e.g. `d3w` or `3l`), via a new operator mode (:issue:`2192`).
-- New ``catpuccin-*`` color themes.
+- New ``catppuccin-*`` color themes.
 
 Improved terminal support
 -------------------------
@@ -1334,7 +1477,7 @@ Deprecations and removed features
 
   Like ``stderr-nocaret``, they will eventually be made read-only.
 - Most ``string`` subcommands no longer append a newline to their input if the input didn't have one (:issue:`8473`, :issue:`3847`)
-- Fish's escape sequence removal (like for ``string length --visible`` or to figure out how wide the prompt is) no longer has special support for non-standard color sequences like from Data General terminals, e.g. the Data General Dasher D220 from 1984. This removes a bunch of work in the common case, allowing ``string length --visible`` to be much faster with unknown escape sequences. We don't expect anyone to have ever used fish with such a terminal (:issue:`8769`).
+- fish's escape sequence removal (like for ``string length --visible`` or to figure out how wide the prompt is) no longer has special support for non-standard color sequences like from Data General terminals, e.g. the Data General Dasher D220 from 1984. This removes a bunch of work in the common case, allowing ``string length --visible`` to be much faster with unknown escape sequences. We don't expect anyone to have ever used fish with such a terminal (:issue:`8769`).
 - Code to upgrade universal variables from fish before 3.0 has been removed. Users who upgrade directly from fish versions 2.7.1 or before will have to set their universal variables & abbreviations again. (:issue:`8781`)
 - The meaning of an empty color variable has changed (:issue:`8793`). Previously, when a variable was set but empty, it would be interpreted as the "normal" color. Now, empty color variables cause the same effect as unset variables - the general highlighting variable for that type is used instead. For example::
 
@@ -1345,14 +1488,14 @@ Deprecations and removed features
 
   This makes it easier to make self-contained color schemes that don't accidentally use color that was set before.
   ``fish_config`` has been adjusted to set known color variables that a theme doesn't explicitly set to empty.
-- ``eval`` is now a reserved keyword, so it can't be used as a function name. This follows ``set`` and ``read``, and is necessary because it can't be cleanly shadowed by a function - at the very least ``eval set -l argv foo`` breaks. Fish will ignore autoload files for it, so left over ``eval.fish`` from previous fish versions won't be loaded.
+- ``eval`` is now a reserved keyword, so it can't be used as a function name. This follows ``set`` and ``read``, and is necessary because it can't be cleanly shadowed by a function - at the very least ``eval set -l argv foo`` breaks. fish will ignore autoload files for it, so left over ``eval.fish`` from previous fish versions won't be loaded.
 - The git prompt in informative mode now defaults to skipping counting untracked files, as this was extremely slow. To turn it on, set :envvar:`__fish_git_prompt_showuntrackedfiles` or set the git config value "bash.showuntrackedfiles" to ``true`` explicitly (which can be done for individual repositories). The "informative+vcs" sample prompt already skipped display of untracked files, but didn't do so in a way that skipped the computation, so it should be quite a bit faster in many cases (:issue:`8980`).
 - The ``__terlar_git_prompt`` function, used by the "Terlar" sample prompt, has been rebuilt as a configuration of the normal ``fish_git_prompt`` to ease maintenance, improve performance and add features (like reading per-repo git configuration). Some slight changes remain; users who absolutely must have the same behavior are encouraged to copy the old function (:issue:`9011`, :issue:`7918`, :issue:`8979`).
 
 Scripting improvements
 ----------------------
 - Quoted command substitution that directly follow a variable expansion (like ``echo "$var$(echo x)"``) no longer affect the variable expansion (:issue:`8849`).
-- Fish now correctly expands command substitutions that are preceded by an escaped dollar (like ``echo \$(echo)``). This regressed in version 3.4.0.
+- fish now correctly expands command substitutions that are preceded by an escaped dollar (like ``echo \$(echo)``). This regressed in version 3.4.0.
 - ``math`` can now handle underscores (``_``) as visual separators in numbers (:issue:`8611`, :issue:`8496`)::
 
     math 5 + 2_123_252
@@ -1372,7 +1515,7 @@ Scripting improvements
 
 Interactive improvements
 ------------------------
-- Fish now reports a special error if a command wasn't found and there is a non-executable file by that name in :envvar:`PATH` (:issue:`8804`).
+- fish now reports a special error if a command wasn't found and there is a non-executable file by that name in :envvar:`PATH` (:issue:`8804`).
 - ``less`` and other interactive commands would occasionally be stopped when run in a pipeline with fish functions; this has been fixed (:issue:`8699`).
 - Case-changing autosuggestions generated mid-token now correctly append only the suffix, instead of duplicating the token (:issue:`8820`).
 - ``ulimit`` learned a number of new options for the resource limits available on Linux, FreeBSD ande NetBSD, and returns a specific warning if the limit specified is not available on the active operating system (:issue:`8823`, :issue:`8786`).
@@ -1382,9 +1525,9 @@ Interactive improvements
 - Since fish 3.2.0, pressing :kbd:`ctrl-d` while a command is running would end up inserting a space into the next commandline, which has been fixed (:issue:`8871`).
 - A bug that caused multi-line prompts to be moved down a line when pasting or switching modes has been fixed (:issue:`3481`).
 - The Web-based configuration system no longer strips too many quotes in the abbreviation display (:issue:`8917`, :issue:`8918`).
-- Fish started with ``--no-config`` will now use the default keybindings (:issue:`8493`)
+- fish started with ``--no-config`` will now use the default keybindings (:issue:`8493`)
 - When fish inherits a :envvar:`USER` environment variable value that doesn't correspond to the current effective user ID, it will now correct it in all cases (:issue:`8879`, :issue:`8583`).
-- Fish sets a new :envvar:`EUID` variable containing the current effective user id (:issue:`8866`).
+- fish sets a new :envvar:`EUID` variable containing the current effective user id (:issue:`8866`).
 - ``history search`` no longer interprets the search term as an option (:issue:`8853`)
 - The status message when a job terminates should no longer be erased by a multiline prompt (:issue:`8817`)
 
@@ -1645,7 +1788,7 @@ Improved terminal support
 
 Other improvements
 ------------------
-- Fish's test suite now uses ``ctest``, and has become much faster to run. It is now also possible to run only specific tests with targets named ``test_$filename`` - ``make test_set.fish`` only runs the set.fish test. (:issue:`7851`)
+- fish's test suite now uses ``ctest``, and has become much faster to run. It is now also possible to run only specific tests with targets named ``test_$filename`` - ``make test_set.fish`` only runs the set.fish test. (:issue:`7851`)
 - The HTML version of the documentation now includes copy buttons for code examples (:issue:`8218`).
 - The HTML version of the documentation and the web-based configuration tool now pick more modern system fonts instead of falling back to Arial and something like Courier New most of the time (:issue:`8632`).
 - The Debian & Ubuntu package linked from fishshell.com is now a single package, rather than split into ``fish`` and ``fish-common`` (:issue:`7845`).
@@ -3244,7 +3387,7 @@ Other fixes and improvements
    variables (:issue:`4200`, :issue:`4341`), executing functions, globs (:issue:`4579`),
    ``string`` reading from standard input (:issue:`4610`), and slicing history
    (in particular, ``$history[1]`` for the last executed command).
--  Fish’s internal wcwidth function has been updated to deal with newer
+-  fish’s internal wcwidth function has been updated to deal with newer
    Unicode, and the width of some characters can be configured via the
    ``fish_ambiguous_width`` (:issue:`5149`) and ``fish_emoji_width`` (:issue:`2652`)
    variables. Alternatively, a new build-time option INTERNAL_WCWIDTH
@@ -3281,7 +3424,7 @@ For distributors and developers
    standard sh instead.
 -  The ``hostname`` command is no longer required for fish to operate.
 
-–
+-
 
 fish 2.7.1 (released December 23, 2017)
 =======================================
@@ -3293,7 +3436,7 @@ session (:issue:`4521`).
 If you are upgrading from version 2.6.0 or before, please also review
 the release notes for 2.7.0 and 2.7b1 (included below).
 
-–
+-
 
 fish 2.7.0 (released November 23, 2017)
 =======================================
@@ -3305,7 +3448,7 @@ from version 2.6.0 or before, please also review the release notes for
 Xcode builds and macOS packages could not be produced with 2.7b1, but
 this is fixed in 2.7.0.
 
-–
+-
 
 fish 2.7b1 (released October 31, 2017)
 ======================================
@@ -4000,13 +4143,13 @@ Backward-incompatible changes
 Other notable fixes and improvements
 ------------------------------------
 
--  Fish no longer silences errors in config.fish (:issue:`2702`)
+-  fish no longer silences errors in config.fish (:issue:`2702`)
 -  Directory autosuggestions will now descend as far as possible if
    there is only one child directory (:issue:`2531`)
 -  Add support for bright colors (:issue:`1464`)
 -  Allow Ctrl-J (``\cj``) to be bound separately from Ctrl-M
    (``\cm``) (:issue:`217`)
--  psub now has a “-s”/“–suffix” option to name the temporary file with
+-  psub now has a “-s”/“-suffix” option to name the temporary file with
    that suffix
 -  Enable 24-bit colors on select terminals (:issue:`2495`)
 -  Support for SVN status in the prompt (:issue:`2582`)
@@ -4030,13 +4173,13 @@ Other notable fixes and improvements
       systemd-analyze, localectl, timedatectl
    -  and more
 
--  Fish no longer has a function called sgrep, freeing it for user
+-  fish no longer has a function called sgrep, freeing it for user
    customization (:issue:`2245`)
 -  A rewrite of the completions for cd, fixing a few bugs (:issue:`2299`, :issue:`2300`,
    :issue:`562`)
 -  Linux VTs now run in a simplified mode to avoid issues (:issue:`2311`)
 -  The vi-bindings now inherit from the emacs bindings
--  Fish will also execute ``fish_user_key_bindings`` when in vi-mode
+-  fish will also execute ``fish_user_key_bindings`` when in vi-mode
 -  ``funced`` will now also check $VISUAL (:issue:`2268`)
 -  A new ``suspend`` function (:issue:`2269`)
 -  Subcommand completion now works better with split /usr (:issue:`2141`)
@@ -4105,7 +4248,7 @@ Other notable fixes and improvements
 
 -  New documentation design (:issue:`1662`), which requires a Doxygen version
    1.8.7 or newer to build.
--  Fish now defines a default directory for other packages to provide
+-  fish now defines a default directory for other packages to provide
    completions. By default this is
    ``/usr/share/fish/vendor-completions.d``; on systems with
    ``pkgconfig`` installed this path is discoverable with
@@ -4396,7 +4539,7 @@ Other Notable Fixes
 -  xsel is no longer built as part of fish. It will still be invoked if
    installed separately :issue:`633`
 -  \__fish_filter_mime no longer spews :issue:`628`
--  The –no-execute option to fish no longer falls over when reaching the
+-  The -no-execute option to fish no longer falls over when reaching the
    end of a block :issue:`624`
 -  fish_config knows how to find fish even if it’s not in the $PATH :issue:`621`
 -  A leading space now prevents writing to history, as is done in bash

CMakeLists.txt

@@ -53,36 +53,36 @@ add_definitions(-DCMAKE_SOURCE_DIR="${REAL_CMAKE_SOURCE_DIR}")
 
 set(build_types Release RelWithDebInfo Debug "")
 if(NOT "${CMAKE_BUILD_TYPE}" IN_LIST build_types)
-  message(WARNING "Unsupported build type ${CMAKE_BUILD_TYPE}. If this doesn't build, try one of Release, RelWithDebInfo or Debug")
+    message(WARNING "Unsupported build type ${CMAKE_BUILD_TYPE}. If this doesn't build, try one of Release, RelWithDebInfo or Debug")
 endif()
 
 add_custom_target(
-  fish ALL
-  COMMAND
-    "${CMAKE_COMMAND}" -E
-      env ${VARS_FOR_CARGO}
-        ${Rust_CARGO}
-          build --bin fish
-          $<$<CONFIG:Release>:--release>
-          $<$<CONFIG:RelWithDebInfo>:--profile=release-with-debug>
-          --target ${Rust_CARGO_TARGET}
-          --no-default-features
-          --features=${FISH_CARGO_FEATURES}
-          ${CARGO_FLAGS}
-    &&
-    "${CMAKE_COMMAND}" -E
-      copy "${rust_target_dir}/${rust_profile}/fish" "${CMAKE_CURRENT_BINARY_DIR}"
-  WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}"
-  USES_TERMINAL
+    fish ALL
+    COMMAND
+        "${CMAKE_COMMAND}" -E
+            env ${VARS_FOR_CARGO}
+                ${Rust_CARGO}
+                    build --bin fish
+                    $<$<CONFIG:Release>:--release>
+                    $<$<CONFIG:RelWithDebInfo>:--profile=release-with-debug>
+                    --target ${Rust_CARGO_TARGET}
+                    --no-default-features
+                    --features=${FISH_CARGO_FEATURES}
+                    ${CARGO_FLAGS}
+        &&
+        "${CMAKE_COMMAND}" -E
+            copy "${rust_target_dir}/${rust_profile}/fish" "${CMAKE_CURRENT_BINARY_DIR}"
+    WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}"
+    USES_TERMINAL
 )
 
 function(CREATE_LINK target)
-  add_custom_target(
-    ${target} ALL
-    DEPENDS fish
-    COMMAND ln -f fish ${target}
-    WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
-  )
+    add_custom_target(
+        ${target} ALL
+        DEPENDS fish
+        COMMAND ln -f fish ${target}
+        WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
+    )
 endfunction(CREATE_LINK)
 
 # Define fish_indent.

CONTRIBUTING.rst

@@ -1,10 +1,10 @@
 ####################
-Contributing To Fish
+Contributing To fish
 ####################
 
 This document tells you how you can contribute to fish.
 
-Fish is free and open source software, distributed under the terms of the GPLv2.
+fish is free and open source software, distributed under the terms of the GPLv2.
 
 Contributions are welcome, and there are many ways to contribute!
 
@@ -21,7 +21,7 @@ Archives are available at https://lists.sr.ht/~krobelus/fish-shell/.
 GitHub
 ======
 
-Fish is available on GitHub, at https://github.com/fish-shell/fish-shell.
+fish is available on GitHub, at https://github.com/fish-shell/fish-shell.
 
 First, you'll need an account there, and you'll need a git clone of fish.
 Fork it on GitHub and then run::
@@ -133,14 +133,14 @@ For formatting, we use:
 - ``fish_indent`` (shipped with fish) for fish script
 - ``ruff format`` for Python
 
-To reformat files, there is a script
+To reformat files, there is an xtask
 
 ::
 
-   build_tools/style.fish --all
-   build_tools/style.fish somefile.rs some.fish
+   cargo xtask format --all
+   cargo xtask format somefile.rs some.fish
 
-Fish Script Style Guide
+fish Script Style Guide
 -----------------------
 
 1. All fish scripts, such as those in the *share/functions* and *tests*
@@ -154,7 +154,7 @@ Fish Script Style Guide
    for public vars or ``_fish`` for private vars to minimize the
    possibility of name clashes with user defined vars.
 
-Configuring Your Editor for Fish Scripts
+Configuring Your Editor for fish Scripts
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 If you use Vim: Install `vim-fish <https://github.com/dag/vim-fish>`__,
@@ -250,39 +250,70 @@ To run all tests and linters, use::
 
     cargo xtask check
 
+.. _localization:
+
 Contributing Translations
 =========================
 
-Fish uses GNU gettext to translate messages from English to other languages.
-We use custom tools for extracting messages from source files and to localize at runtime.
-This means that we do not have a runtime dependency on the gettext library.
-It also means that some features are not supported, such as message context and plurals.
-We also expect all files to be UTF-8-encoded.
-In practice, this should not matter much for contributing translations.
+fish can localize messages present in its Rust source code,
+as well as messages in the various fish scripts present in this repository.
+The latter include a large amount of automatically identified messages,
+originating for example from fish function descriptions.
+When translating, prioritize the messages from the Rust source code.
 
-Translation sources are stored in the ``localization/po`` directory and named ``ll_CC.po``,
+fish uses two different localization systems:
+`GNU gettext <https://www.gnu.org/software/gettext/>`__ and `Fluent <https://projectfluent.org/>`__.
+The former is used for all messages from fish scripts.
+For messages from the Rust source code, we are in the process of replacing gettext with Fluent.
+At the moment, both are used side-by-side,
+with some messages localized with gettext and the others with Fluent.
+
+We use custom tools for extracting messages from source files and for gettext localization at runtime.
+This means that we do not have a runtime dependency on the gettext library.
+It also means that some of gettext's features are not supported, such as message context and plurals.
+We expect all files to be UTF-8-encoded.
+
+Translation sources for gettext are stored in the ``localization/po`` directory and named ``ll_CC.po``,
+whereas Fluent uses the ``localization/fluent`` directory and names of the shape ``ll_CC.ftl``,
 where ``ll`` is the two (or possibly three) letter ISO 639-1 language code of the target language
-(e.g. ``pt`` for Portuguese). ``CC`` is an ISO 3166 country/territory code,
-(e.g. ``BR`` for Brazil).
-An example for a valid name is ``pt_BR.po``, indicating Brazilian Portuguese.
+(e.g. ``pt`` for Portuguese).
+``CC`` is an ISO 3166 country/territory code, (e.g. ``BR`` for Brazil).
+An example for a valid name is ``pt_BR.po`` for gettext and ``pt_BR.ftl`` for Fluent,
+indicating Brazilian Portuguese.
 These are the files you will interact with when adding translations.
 
+In some cases, we also use language identifiers without a county code, i.e. ``ll.po``/``ll.ftl``.
+Which variant is chosen involves various trade-offs for fallback behavior.
+If you want to add a new language or language variant, feel free to ask about this.
+Generally, if people who understand any variant of the language
+are expected to understand the version you add
+and there are no existing translations for another variant of the language,
+it probably makes sense to omit the country code,
+otherwise to use it for all variants of the language.
+
 Adding translations for a new language
 --------------------------------------
 
-Creating new translations requires the Gettext tools.
-More specifically, you will need ``msguniq`` and ``msgmerge`` for creating translations for a new
-language.
-To create a new translation, run::
+Creating new translations for gettext requires the gettext tools.
+More specifically, you will need ``msguniq``, ``msgmerge``, and ``msgmerge``
+for creating translations for a new language.
+To create a PO file for a new language ``ll_CC``, run::
 
-    build_tools/update_translations.fish localization/po/ll_CC.po
+    cargo xtask gettext new ll_CC
 
-This will create a new PO file containing all messages available for translation.
+This will create a new PO file in ``localization/po/``
+containing all messages available for translation.
 If the file already exists, it will be updated.
 
-After modifying a PO file, you can recompile fish, and it will integrate the modifications you made.
+For Fluent, it is sufficient to create a new, empty file
+with the language-appropriate name in ``localization/fluent``.
+
+After modifying a translation file, you can recompile fish,
+and it will integrate the modifications you made.
 This requires that the ``msgfmt`` utility is installed (comes as part of ``gettext``).
-It is important that the ``localize-messages`` cargo feature is enabled, which it is by default.
+For messages localized with Fluent, recompiling fish is not necessary when you use a debug build.
+Then, restarting fish is sufficient for seeing updated translations.
+It is important that the ``localize-messages`` Cargo feature is enabled, which it is by default.
 You can explicitly enable it using::
 
     cargo build --features=localize-messages
@@ -295,7 +326,26 @@ or within the running fish shell::
 
     set LANG pt_BR.utf8
 
-For more options regarding how to choose languages, see
+Alternatively, you can also use the built-in ``status language`` command, e.g.::
+
+    status language set pt_BR
+
+Use
+
+::
+
+    status language list-available
+
+to see a list of the available language identifiers.
+This might also be helpful for checking that your new translations are recognized as expected.
+Note that using environment variables enables fallback behavior,
+e.g. if you specify ``LANG=de_DE.utf8`` and we do not have a ``de_DE`` catalog but a ``de`` catalog,
+you will see messages from the latter.
+With ``status language``, only exact matches are supported,
+giving you more control over the fallback order.
+If ``status language`` is used, it overrides the environment variable configuration.
+
+For more options regarding how to choose languages via environment variables, see
 `the corresponding gettext documentation
 <https://www.gnu.org/software/gettext/manual/html_node/Locale-Environment-Variables.html>`__.
 One neat thing you can do is set a list of languages to check for translations in the order defined
@@ -303,14 +353,19 @@ using the ``LANGUAGE`` variable, e.g.::
 
     set LANGUAGE pt_BR de_DE
 
+or using::
+
+    status language set pt_BR de
+
 to try to translate messages to Portuguese, if that fails try German, and if that fails too you will
-see the English version defined in the source code.
+see the default English version.
 
 Modifying existing translations
 -------------------------------
 
-If you want to work on translations for a language which already has a corresponding ``po`` file, it
-is sufficient to edit this file. No other changes are necessary.
+If you want to work on translations for a language which already has translations, it
+is sufficient to edit the existing files.
+No other changes are necessary.
 
 After recompiling fish, you should be able to see your translations in action. See the previous
 section for details.
@@ -334,7 +389,7 @@ For example::
     msgid "%s: No suitable job\n"
     msgstr "%s: Inget passande jobb\n"
 
-Any ``%s`` or ``%d`` are placeholders that fish will use for formatting at runtime. It is important that they match - the translated string should have the same placeholders in the same order.
+Any ``%s`` or ``%d`` are placeholders that fish will use for formatting at runtime. It is important that they match - the translated string must have the same placeholders in the same order.
 
 Also any escaped characters, like that ``\n`` newline at the end, should be kept so the translation has the same behavior.
 
@@ -343,30 +398,166 @@ Our tests run ``msgfmt --check-format /path/to/file``, so they would catch misma
 Be cautious about blindly updating an existing translation file.
 ``msgid`` strings should never be updated manually, only by running the appropriate script.
 
-Modifications to strings in source files
-----------------------------------------
+Editing FTL files
+-----------------
+
+To get familiar with Fluent's FTL format,
+you can read `Fluent's guide <https://projectfluent.org/fluent/guide/>`__.
+
+The core principle is that each message has an ID.
+This ID is specified in the source code.
+At runtime, Fluent checks FTL files according to the user's language settings
+to try to map the ID to a localized message.
+All messages are localized into English because the source code only contains IDs, not proper messages.
+Check ``localization/fluent/en.ftl`` to see which IDs are in use and what the corresponding messages are.
+
+Some messages receive arguments, called variables in Fluent.
+In Fluent, each variable has a name, which allows reordering them,
+so they can appear in the order which makes most sense for the language.
+The general format of a variable in an FTL file is ``{ $variable_name }``.
+
+The Fluent ecosystem is not as mature as gettext, meaning there is less available tooling.
+Therefore, we provide some of our own tools.
+Running ``cargo xtask fluent`` provides an overview.
+
+For translators, the following can be useful::
+
+    cargo xtask fluent format
+
+to make FTL files conform to our expected format,
+
+::
+
+    cargo xtask fluent show-missing
+
+to show which message IDs do not have a translation yet, and
+
+::
+
+    cargo xtask fluent check
+
+to run checks on the FTL files, which can catch some mistakes.
+Each of these commands takes optional path arguments,
+so if you are working on a certain file like ``pt_BR.ftl``,
+you might want to use
+
+::
+
+    cargo xtask fluent check localization/fluent/pt_BR.ftl
+
+from the repository root directory,
+or, if you are in the ``localization/fluent`` directory,
+
+::
+
+    cargo xtask fluent check pt_BR.ftl
+
+If you want formatting in your editor,
+
+::
+
+    cargo --quiet xtask fluent format -
+
+might be useful, which reads FTL text from stdin and writes a formatted version to stdout,
+or a copy of stdin if formatting failed.
+Instead of invoking Cargo each time, you could also invoke the ``xtask`` binary if it exists.
+
+A simple format-on-write setup in Vim:
+
+.. code:: vim
+
+    function FormatFTL()
+        let cursor = getpos('.')
+        :%!cargo --quiet xtask fluent format -
+        call setpos('.', cursor)
+    endfunction
+
+    augroup ftl
+        autocmd!
+        autocmd! BufWritePre *.ftl :call FormatFTL()
+    augroup END
+
+or equivalently in Lua for NeoVim:
+
+.. code:: lua
+
+    local augroup_ftl = vim.api.nvim_create_augroup("ftl", { clear = true })
+    vim.api.nvim_create_autocmd("BufWritePre", {
+        group = augroup_ftl,
+        pattern = "*.ftl",
+        callback = function()
+            local cursor = vim.fn.getpos(".")
+            vim.cmd("%!cargo --quiet xtask fluent format -")
+            vim.fn.setpos(".", cursor)
+        end,
+    })
+
+There is also a `Vim plugin <https://github.com/projectfluent/fluent.vim>`__ for syntax highlighting.
+
+Modifications to strings in source files (gettext-only)
+-------------------------------------------------------
 
 If a string changes in the sources, the old translations will no longer work.
-They will be preserved in the PO files, but commented-out (starting with ``#~``).
 If you add/remove/change a translatable strings in a source file,
-run ``build_tools/update_translations.fish`` to propagate this to all translation files (``localization/po/*.po``).
+run ``cargo xtask gettext update`` to propagate this to all translation files (``localization/po/*.po``).
 This is only relevant for developers modifying the source files of fish or fish scripts.
+Note translations for messages which are no longer present in the sources will be deleted from the PO files.
+If the source string changed in a way which should not affect translations,
+consider updating the ``msgid`` in the PO files such that translations are preserved.
 
 Setting Code Up For Translations
 --------------------------------
 
-All non-debug messages output for user consumption should be marked for
-translation. In Rust, this requires the use of the ``wgettext!`` or ``wgettext_fmt!``
-macros:
+All non-debug messages output for user consumption should be marked for translation.
+In Rust, this requires the use of the ``localize!`` macro for Fluent localization, e.g.:
+
+.. code:: rust
+
+    localize!(
+        "some-message-id" = "English version of the message. Must be a valid Fluent message definition. Example variables: { $var1 }, { $var2 }",
+        var1 = "some string",
+        var2 = 42,
+    );
+
+where the first key-value pair is the message's Fluent ID and the English version of the message.
+The remaining key-value pairs specify Fluent variables and their values.
+These must match the variables used in the message definition.
+
+For changing message IDs or associated variable names accross all FTL files, the
 
 ::
 
-    streams.out.append(wgettext_fmt!("%s: There are no jobs\n", argv[0]));
+    cargo xtask fluent rename
 
-All messages in fish script must be enclosed in single or double quote
-characters for our message extraction script to find them.
-They must also be translated via a command substitution. This means
-that the following are **not** valid:
+command can be helpful.
+The definitions in the Rust sources are the source of truth for the English version of the messages.
+Our tooling automatically generates a corresponding ``en.po`` file from these definitions,
+but that file is fully auto-generated and manual modifications to it are not supported.
+The `en.po` file exists to allow using Fluent tooling which expects such a file,
+and to be able to detect changes to the definitions in the Rust sources.
+Note that changes to the message definitions have consequences for translations.
+Our tooling automatically detects such changes.
+In some cases, they can be resolved automatically,
+e.g. when a message ID no longer exists, the translations will be deleted.
+If no automatic resolution is possible, annotations will be added to the affected translations,
+indicating that they need developer attention.
+As long as such annotations are present, our checks will not pass.
+To resolve them, use ``cargo xtask fluent resolve-outdated``,
+or, for languages into which you can translate,
+update the translation and remove the annotation manually.
+
+Legacy gettext localization uses the ``wgettext!`` or ``wgettext_fmt!`` macros.
+New code should use Fluent instead.
+
+.. code:: rust
+
+    streams.out.appendln(&wgettext_fmt!("%s: There are no jobs", argv[0]));
+
+For explicit localization in fish scripts,
+all messages must be enclosed in single or double quote characters
+for our message extraction script to find them.
+They must also be translated via a command substitution.
+This means that the following are **not** valid:
 
 ::
 

Cargo.toml

@@ -2,80 +2,98 @@
 members = ["crates/*"]
 
 [workspace.package]
+edition = "2024"
 # To build revisions that use Corrosion (those before 2024-01), use CMake 3.19, Rustc 1.78 and Rustup 1.27.
 rust-version = "1.85"
-edition = "2024"
 repository = "https://github.com/fish-shell/fish-shell"
 # see doc_src/license.rst for details
 # don't forget to update COPYING and debian/copyright too
 license = "GPL-2.0-only AND LGPL-2.0-or-later AND MIT AND PSF-2.0"
 
 [workspace.dependencies]
+anstyle = "1.0.13"
+anyhow = "1.0.102"
 assert_matches = "1.5.0"
 bitflags = "2.5.0"
 cc = "1.0.94"
 cfg-if = "1.0.3"
 clap = { version = "4.5.54", features = ["derive"] }
+clap_complete = { version = "4.6.4", features = ["unstable-dynamic"] }
 errno = "0.3.0"
 fish-build-helper = { path = "crates/build-helper" }
 fish-build-man-pages = { path = "crates/build-man-pages" }
 fish-color = { path = "crates/color" }
 fish-common = { path = "crates/common" }
 fish-fallback = { path = "crates/fallback" }
+fish-feature-flags = { path = "crates/feature-flags" }
+fish-fluent = { path = "crates/fluent" }
+fish-fluent-extraction = { path = "crates/fluent-extraction" }
 fish-gettext = { path = "crates/gettext" }
 fish-gettext-extraction = { path = "crates/gettext-extraction" }
 fish-gettext-maps = { path = "crates/gettext-maps" }
 fish-gettext-mo-file-parser = { path = "crates/gettext-mo-file-parser" }
+fish-localization = { path = "crates/localization" }
+fish-localization-extraction = { path = "crates/localization-extraction" }
 fish-printf = { path = "crates/printf", features = ["widestring"] }
 fish-tempfile = { path = "crates/tempfile" }
 fish-util = { path = "crates/util" }
 fish-wcstringutil = { path = "crates/wcstringutil" }
+fish-wgetopt = { path = "crates/wgetopt" }
 fish-widecharwidth = { path = "crates/widecharwidth" }
 fish-widestring = { path = "crates/widestring" }
-fish-wgetopt = { path = "crates/wgetopt" }
+fluent = { git = "https://github.com/danielrainer/fluent-rs", rev = "cf712bced280b217b6307edabc2089b3e57204ab" }
+fluent-ftl-tools = { git = "https://codeberg.org/danielrainer/fluent-ftl-tools", rev = "5917664c8f2e4928ef1e480ff5c13bbe1e226066" }
+fluent-syntax = { git = "https://github.com/danielrainer/fluent-rs", rev = "cf712bced280b217b6307edabc2089b3e57204ab" }
+ignore = "0.4.25"
 itertools = "0.14.0"
 libc = "0.2.177"
 # lru pulls in hashbrown by default, which uses a faster (though less DoS resistant) hashing algo.
 # disabling default features uses the stdlib instead, but it doubles the time to rewrite the history
 # files as of 22 April 2024.
-lru = "0.16.2"
+lru = "0.18.0"
 nix = { version = "0.31.1", default-features = false, features = [
-    "event",
-    "fs",
-    "inotify",
-    "hostname",
-    "resource",
-    "process",
-    "signal",
-    "term",
-    "user",
+  "event",
+  "fs",
+  "hostname",
+  "inotify",
+  "process",
+  "resource",
+  "signal",
+  "term",
+  "user",
 ] }
 num-traits = "0.2.19"
 once_cell = "1.19.0"
 pcre2 = { git = "https://github.com/fish-shell/rust-pcre2", tag = "0.2.9-utf32", default-features = false, features = [
-    "utf32",
+  "utf32",
 ] }
 phf = { version = "0.13", default-features = false }
+phf_shared = "0.13"
 phf_codegen = "0.13"
 portable-atomic = { version = "1", default-features = false, features = [
-    "fallback",
+  "fallback",
 ] }
 proc-macro2 = "1.0"
-rand = { version = "0.9.2", default-features = false, features = [
-    "small_rng",
-    "thread_rng",
+rand = { version = "0.10.1", default-features = false, features = [
+  "thread_rng",
 ] }
+regex = "1.12.3"
 rsconf = "0.3.0"
 rust-embed = { version = "8.11.0", features = [
-    "deterministic-timestamps",
-    "include-exclude",
-    "interpolate-folder-path",
+  "deterministic-timestamps",
+  "include-exclude",
+  "interpolate-folder-path",
 ] }
+rustc_version = "0.4.1"
 serial_test = { version = "3", default-features = false }
-widestring = "1.2.0"
+strum_macros = "0.28.0"
+syn = { version = "2.0.117", default-features = false, features = ["parsing"] }
+unic-langid = "0.9.6"
 unicode-segmentation = "1.12.0"
 unicode-width = "0.2.0"
 unix_path = "1.0.1"
+walkdir = "2.5.0"
+widestring = "1.2.0"
 xterm-color = "1.0.1"
 
 [profile.release]
@@ -88,7 +106,7 @@ debug = true
 
 [package]
 name = "fish"
-version = "4.5.0"
+version = "4.7.1"
 edition.workspace = true
 rust-version.workspace = true
 default-run = "fish"
@@ -106,8 +124,12 @@ fish-build-man-pages = { workspace = true, optional = true }
 fish-color.workspace = true
 fish-common.workspace = true
 fish-fallback.workspace = true
+fish-feature-flags.workspace = true
+fish-fluent.workspace = true
+fish-fluent-extraction = { workspace = true, optional = true }
 fish-gettext = { workspace = true, optional = true }
 fish-gettext-extraction = { workspace = true, optional = true }
+fish-localization = { workspace = true, optional = true }
 fish-printf.workspace = true
 fish-tempfile.workspace = true
 fish-util.workspace = true
@@ -115,6 +137,7 @@ fish-wcstringutil.workspace = true
 fish-wgetopt.workspace = true
 fish-widecharwidth.workspace = true
 fish-widestring.workspace = true
+fluent.workspace = true
 itertools.workspace = true
 libc.workspace = true
 lru.workspace = true
@@ -124,7 +147,7 @@ num-traits.workspace = true
 once_cell.workspace = true
 pcre2.workspace = true
 rand.workspace = true
-unicode-width.workspace = true
+strum_macros.workspace = true
 xterm-color.workspace = true
 
 [target.'cfg(not(target_has_atomic = "64"))'.dependencies]
@@ -132,16 +155,16 @@ portable-atomic.workspace = true
 
 [target.'cfg(windows)'.dependencies]
 rust-embed = { workspace = true, features = [
-    "deterministic-timestamps",
-    "debug-embed",
-    "include-exclude",
-    "interpolate-folder-path",
+  "deterministic-timestamps",
+  "debug-embed",
+  "include-exclude",
+  "interpolate-folder-path",
 ] }
 [target.'cfg(not(windows))'.dependencies]
 rust-embed = { workspace = true, features = [
-    "deterministic-timestamps",
-    "include-exclude",
-    "interpolate-folder-path",
+  "deterministic-timestamps",
+  "include-exclude",
+  "interpolate-folder-path",
 ] }
 
 [dev-dependencies]
@@ -153,6 +176,7 @@ fish-build-helper.workspace = true
 fish-gettext-mo-file-parser.workspace = true
 phf_codegen = { workspace = true, optional = true }
 rsconf.workspace = true
+rustc_version.workspace = true
 
 [target.'cfg(windows)'.build-dependencies]
 unix_path.workspace = true
@@ -177,22 +201,28 @@ path = "src/bin/fish_key_reader.rs"
 default = ["embed-manpages", "localize-messages"]
 benchmark = []
 embed-manpages = ["dep:fish-build-man-pages"]
-# Enable gettext localization at runtime. Requires the `msgfmt` tool to generate catalog data at
-# build time.
-localize-messages = ["dep:fish-gettext"]
+# This feature is used to enable extracting Fluent IDs from the source code for localization check.
+# For normal builds, it should be disabled.
+# It only needs to be enabled if checking Fluent Translation List (FTL) files is desired.
+fluent-extract = ["fish-fluent/fluent-extract", "dep:fish-fluent-extraction"]
 # This feature is used to enable extracting messages from the source code for localization.
 # It only needs to be enabled if updating these messages (and the corresponding PO files) is
-# desired. This happens when running tests via `cargo xtask check` and when calling
-# `build_tools/update_translations.fish`, so there should not be a need to enable it manually.
+# desired. This happens for the `gettext` xtask, which is also invoked via `cargo xtask check`.
+# There should not be a need to enable this feature manually.
 gettext-extract = ["dep:fish-gettext-extraction"]
+# Enable gettext localization at runtime. Requires the `msgfmt` tool to generate catalog data at
+# build time.
+localize-messages = [
+  "fish-fluent/localize-messages",
+  "dep:fish-gettext",
+  "dep:fish-localization",
+]
 
 # The following features are auto-detected by the build-script and should not be enabled manually.
 tsan = []
 
 [workspace.lints]
-rust.non_camel_case_types = "allow"
-rust.non_upper_case_globals = "allow"
-rust.unknown_lints = "allow"
+rust.unknown_lints = { level = "allow", priority = -1 }
 rust.unstable_name_collisions = "allow"
 rustdoc.private_intra_doc_links = "allow"
 
@@ -212,6 +242,7 @@ needless_lifetimes = "allow"
 new_without_default = "allow"
 option_map_unit_fn = "allow"
 ptr_offset_by_literal = "warn"
+redundant_clone = "warn"
 ref_option = "warn"
 semicolon_if_nothing_returned = "warn"
 stable_sort_primitive = "warn"
@@ -224,6 +255,8 @@ unused_trait_names = "warn"
 # In the future, they might change to flag other methods of printing.
 print_stdout = "deny"
 print_stderr = "deny"
+# usage in tests is fine since it avoids interacting with TopicMonitor
+# and is configured in clippy.toml with `allow-print-in-tests`
 
 [lints]
 workspace = true

README.rst

@@ -1,9 +1,5 @@
-.. |Cirrus CI| image:: https://api.cirrus-ci.com/github/fish-shell/fish-shell.svg?branch=master
-      :target: https://cirrus-ci.com/github/fish-shell/fish-shell
-      :alt: Cirrus CI Build Status
-
-`fish <https://fishshell.com/>`__ - the friendly interactive shell |Build Status| |Cirrus CI|
-=============================================================================================
+`fish <https://fishshell.com/>`__ - the friendly interactive shell |Build Status|
+=================================================================================
 
 fish is a smart and user-friendly command line shell for macOS, Linux,
 and the rest of the family. fish includes features like syntax
@@ -31,7 +27,7 @@ macOS
 
 fish can be installed:
 
--  using `Homebrew <http://brew.sh/>`__: ``brew install fish``
+-  using `Homebrew <https://brew.sh/>`__: ``brew install fish``
 -  using `MacPorts <https://www.macports.org/>`__:
    ``sudo port install fish``
 -  using the `installer from fishshell.com <https://fishshell.com/>`__
@@ -66,7 +62,7 @@ Windows
    for Linux with the instructions for the appropriate distribution
    listed above under “Packages for Linux”, or from source with the
    instructions below.
--  Fish can also be installed on all versions of Windows using
+-  fish can also be installed on all versions of Windows using
    `Cygwin <https://cygwin.com/>`__ or `MSYS2 <https://github.com/Berrysoft/fish-msys2>`__.
 
 Building from source
@@ -221,5 +217,5 @@ There is also a fish tag on Stackoverflow, but it is typically a poor fit.
 Found a bug? Have an awesome idea? Please `open an
 issue <https://github.com/fish-shell/fish-shell/issues/new>`__.
 
-.. |Build Status| image:: https://github.com/fish-shell/fish-shell/workflows/make%20test/badge.svg
-   :target: https://github.com/fish-shell/fish-shell/actions
+.. |Build Status| image:: https://github.com/fish-shell/fish-shell/actions/workflows/test.yml/badge.svg
+   :target: https://github.com/fish-shell/fish-shell/actions/workflows/test.yml

build.rs

@@ -6,6 +6,10 @@
 use std::path::{Path, PathBuf};
 
 fn main() {
+    let is_nightly =
+        rustc_version::version_meta().unwrap().channel == rustc_version::Channel::Nightly;
+    rsconf::declare_cfg("nightly", is_nightly);
+
     setup_paths();
 
     // Add our default to enable tools that don't go through CMake, like "cargo test" and the

build_tools/check.sh

@@ -8,11 +8,29 @@ if [ "$FISH_CHECK_LINT" = false ]; then
     lint=false
 fi
 
+case "$(uname)" in
+    MSYS*)
+        is_cygwin=true
+        cygwin_var=MSYS
+        ;;
+    CYGWIN*)
+        is_cygwin=true
+        cygwin_var=CYGWIN
+        ;;
+    *)
+        is_cygwin=false
+        ;;
+esac
+
 check_dependency_versions=false
 if [ "${FISH_CHECK_DEPENDENCY_VERSIONS:-false}" != false ]; then
     check_dependency_versions=true
 fi
 
+green='\e[0;32m'
+yellow='\e[1;33m'
+reset='\e[m'
+
 if $check_dependency_versions; then
     command -v curl
     command -v jq
@@ -42,7 +60,11 @@ cargo() {
     fi
 }
 
+# shellcheck disable=2317,2329
 cleanup () {
+    if [ -n "$fluent_extraction_dir" ] && [ -e "$fluent_extraction_dir" ]; then
+        rm -r "$fluent_extraction_dir"
+    fi
     if [ -n "$gettext_template_dir" ] && [ -e "$gettext_template_dir" ]; then
         rm -r "$gettext_template_dir"
     fi
@@ -69,26 +91,84 @@ if [ -n "$FISH_TEST_MAX_CONCURRENCY" ]; then
     export CARGO_BUILD_JOBS="$FISH_TEST_MAX_CONCURRENCY"
 fi
 
+fluent_extraction_dir=$(mktemp -d)
 gettext_template_dir=$(mktemp -d)
 (
+    # shellcheck disable=2030
+    export FISH_FLUENT_EXTRACTION_DIR="$fluent_extraction_dir"
+    # shellcheck disable=2030
     export FISH_GETTEXT_EXTRACTION_DIR="$gettext_template_dir"
-    cargo build --workspace --all-targets --features=gettext-extract
+    cargo build --workspace --all-targets --features=fluent-extract,gettext-extract
 )
 if $lint; then
     if command -v cargo-deny >/dev/null; then
         cargo deny --all-features --locked --exclude-dev check licenses
     fi
-    PATH="$build_dir:$PATH" "$workspace_root/build_tools/style.fish" --all --check
-    for features in "" --no-default-features; do
+
+    if command -v shellcheck >/dev/null || { test -n "$CI" && ! $is_cygwin; }; then
+        cargo xtask shellcheck
+    fi
+
+    PATH="$build_dir:$PATH" cargo xtask format --all --check
+    for features in "" --no-default-features --all-features; do
         cargo clippy --workspace --all-targets $features
     done
+
+    cargo xtask gettext --rust-extraction-dir="$gettext_template_dir" check
 fi
-cargo test --no-default-features --workspace --all-targets
+
+# An outdated `en.ftl` or translations using variables not provided by the source could crash fish,
+# so don't treat this as a lint.
+cargo xtask fluent check --from-source="$fluent_extraction_dir"
+
+# When running `cargo test`, some binaries (e.g. `fish_gettext_extraction`)
+# are dynamically linked against Rust's `std-xxx.dll` instead of being
+# statically link as they usually are.
+# On Cygwin, `PATH`is not properly updated to point to the `std-xxx.dll`
+# location, so we have to do it manually.
+# See:
+# - https://github.com/rust-lang/rust/issues/149050
+# - https://github.com/msys2/MSYS2-packages/issues/5784
+(
+    if $is_cygwin; then
+        PATH="$PATH:$(rustc --print target-libdir)"
+        export PATH
+    fi
+    # shellcheck disable=2031
+    export FISH_FLUENT_EXTRACTION_DIR="$fluent_extraction_dir"
+    cargo test --no-default-features --workspace --all-targets
+)
 cargo test --doc --workspace
+
 if $lint; then
     cargo doc --workspace --no-deps
 fi
-FISH_GETTEXT_EXTRACTION_DIR=$gettext_template_dir "$workspace_root/tests/test_driver.py" "$build_dir"
+
+system_tests() {
+    "$workspace_root/tests/test_driver.py" "$build_dir" "$@"
+}
+
+if $is_cygwin; then
+    # shellcheck disable=2059
+    printf "=== Running ${green}integration tests ${yellow}with${green} symlinks${reset}\n"
+    (
+        export "$cygwin_var"=winsymlinks
+        system_tests
+    )
+
+    # shellcheck disable=2059
+    printf "=== Running ${green}integration tests ${yellow}without${green} symlinks${reset}\n"
+    (
+        # Only redo the tests that use `ln` to saves some time
+        export "$cygwin_var"=
+        # shellcheck disable=2046
+        system_tests $(grep -l -E '\bln\b' -r tests/checks/)
+    )
+else
+    # shellcheck disable=2059
+    printf "=== Running ${green}integration tests${reset}\n"
+    system_tests
+fi
 
 exit
 }

build_tools/fish_xgettext.fish

@@ -1,147 +0,0 @@
-#!/usr/bin/env fish
-#
-# Tool to generate gettext messages template file.
-# Writes to stdout.
-# Intended to be called from `update_translations.fish`.
-
-argparse use-existing-template= -- $argv
-or exit $status
-
-begin
-    # Write header. This is required by msguniq.
-    # Note that this results in the file being overwritten.
-    # This is desired behavior, to get rid of the results of prior invocations
-    # of this script.
-    begin
-        echo 'msgid ""'
-        echo 'msgstr ""'
-        echo '"Content-Type: text/plain; charset=UTF-8\n"'
-        echo ""
-    end
-
-    set -g workspace_root (path resolve (status dirname)/..)
-
-    set -l rust_extraction_dir
-    if set -l --query _flag_use_existing_template
-        set rust_extraction_dir $_flag_use_existing_template
-    else
-        set rust_extraction_dir (mktemp -d)
-        # We need to build to ensure that the proc macro for extracting strings runs.
-        FISH_GETTEXT_EXTRACTION_DIR=$rust_extraction_dir cargo check --features=gettext-extract
-        or exit 1
-    end
-
-    function mark_section
-        set -l section_name $argv[1]
-        echo 'msgid "fish-section-'$section_name'"'
-        echo 'msgstr ""'
-        echo ''
-    end
-
-    mark_section tier1-from-rust
-
-    # Get rid of duplicates and sort.
-    find $rust_extraction_dir -type f -exec cat {} + | msguniq --no-wrap --sort-output
-    or exit 1
-
-    if not set -l --query _flag_use_existing_template
-        rm -r $rust_extraction_dir
-    end
-
-    function extract_fish_script_messages_impl
-        set -l regex $argv[1]
-        set -e argv[1]
-        # Using xgettext causes more trouble than it helps.
-        # This is due to handling of escaping in fish differing from formats xgettext understands
-        # (e.g. POSIX shell strings).
-        # We work around this issue by manually writing the file content.
-
-        # Steps:
-        # 1. We extract strings to be translated from the relevant files and drop the rest. This step
-        #    depends on the regex matching the entire line, and the first capture group matching the
-        #    string.
-        # 2. We unescape. This gets rid of some escaping necessary in fish strings.
-        # 3. The resulting strings are sorted alphabetically. This step is optional. Not sorting would
-        #    result in strings from the same file appearing together. Removing duplicates is also
-        #    optional, since msguniq takes care of that later on as well.
-        # 4. Single backslashes are replaced by double backslashes. This results in the backslashes
-        #    being interpreted as literal backslashes by gettext tooling.
-        # 5. Double quotes are escaped, such that they are not interpreted as the start or end of
-        #    a msgid.
-        # 6. We transform the string into the format expected in a PO file.
-        cat $argv |
-            string replace --filter --regex $regex '$1' |
-            string unescape |
-            sort -u |
-            sed -E -e 's_\\\\_\\\\\\\\_g' -e 's_"_\\\\"_g' -e 's_^(.*)$_msgid "\1"\nmsgstr ""\n_'
-    end
-
-    function extract_fish_script_messages
-        set -l tier $argv[1]
-        set -e argv[1]
-        if not set -q argv[1]
-            return
-        end
-        # This regex handles explicit requests to translate a message. These are more important to translate
-        # than messages which should be implicitly translated.
-        set -l explicit_regex '.*\( *_ (([\'"]).+?(?<!\\\\)\\2) *\).*'
-        mark_section "$tier-from-script-explicitly-added"
-        extract_fish_script_messages_impl $explicit_regex $argv
-
-        # This regex handles descriptions for `complete` and `function` statements. These messages are not
-        # particularly important to translate. Hence the "implicit" label.
-        set -l implicit_regex '^(?:\s|and |or )*(?:complete|function).*? (?:-d|--description) (([\'"]).+?(?<!\\\\)\\2).*'
-        mark_section "$tier-from-script-implicitly-added"
-        extract_fish_script_messages_impl $implicit_regex $argv
-    end
-
-    set -g share_dir $workspace_root/share
-
-    set -l tier1 $share_dir/config.fish
-    set -l tier2
-    set -l tier3
-
-    for file in $share_dir/completions/*.fish $share_dir/functions/*.fish
-        # set -l tier (string match -r '^# localization: .*' <$file)
-        set -l tier (string replace -rf -m1 \
-            '^# localization: (.*)$' '$1' <$file)
-        if set -q tier[1]
-            switch "$tier"
-                case tier1 tier2 tier3
-                    set -a $tier $file
-                case 'skip*'
-                case '*'
-                    echo >&2 "$file:1 unexpected localization tier: $tier"
-                    exit 1
-            end
-            continue
-        end
-        set -l dirname (path basename (path dirname $file))
-        set -l command_name (path basename --no-extension $file)
-        if test $dirname = functions &&
-                string match -q -- 'fish_*' $command_name
-            set -a tier1 $file
-            continue
-        end
-        if test $dirname != completions
-            echo >&2 "$file:1 missing localization tier for function file"
-            exit 1
-        end
-        if test -e $workspace_root/doc_src/cmds/$command_name.rst
-            set -a tier1 $file
-        else
-            set -a tier3 $file
-        end
-    end
-
-    extract_fish_script_messages tier1 $tier1
-    extract_fish_script_messages tier2 $tier2
-    extract_fish_script_messages tier3 $tier3
-end |
-    # At this point, all extracted strings have been written to stdout,
-    # starting with the ones taken from the Rust sources,
-    # followed by strings explicitly marked for translation in fish scripts,
-    # and finally the strings from fish scripts which get translated implicitly.
-    # Because we do not eliminate duplicates across these categories,
-    # we do it here, since other gettext tools expect no duplicates.
-    msguniq --no-wrap

build_tools/lsan_suppressions.txt

@@ -0,0 +1 @@
+leak:fish_fluent::AVAILABLE_LANGUAGES

build_tools/make_linux_packages.sh

@@ -0,0 +1,83 @@
+#!/bin/sh
+
+# This script takes a source tarball (from build_tools/make_tarball.sh) and a vendor tarball (from
+# build_tools/make_vendor_tarball.sh, generated if not present), and produces:
+# * Appropriately-named symlinks to look like a Debian package
+# * Debian .changes and .dsc files with plain names ($version-1) and supported Ubuntu prefixes
+#   ($version-1~somedistro)
+# * An RPM spec file
+# By default, input and output files go in ~/fish_built, but this can be controlled with the
+# FISH_ARTEFACT_PATH environment variable.
+
+{
+
+set -e
+
+version=$1
+[ -n "$version" ] || { echo "Version number required as argument" >&2; exit 1; }
+
+
+[ -n "$DEB_SIGN_KEYID$DEB_SIGN_KEYFILE" ] ||
+    echo "Warning: neither DEB_SIGN_KEYID or DEB_SIGN_KEYFILE environment variables are set; you
+will need a signing key for the author of the most recent debian/changelog entry." >&2
+
+workpath=${FISH_ARTEFACT_PATH:-~/fish_built}
+source_tarball="$workpath"/fish-"$version".tar.xz
+vendor_tarball="$workpath"/fish-"$version"-vendor.tar.xz
+
+[ -e "$source_tarball" ] || { echo "Missing source tarball, expected at $source_tarball" >&2; exit 1; }
+cd "$workpath"
+
+# Unpack the sources
+tar xf "$source_tarball"
+sourcepath="$workpath"/fish-"$version"
+
+# Generate the vendor tarball if it is not already present
+[ -e "$vendor_tarball" ] || (cd "$sourcepath"; build_tools/make_vendor_tarball.sh;)
+
+# This step requires network access, so do it early in case it fails
+# sh has no real array support
+ubuntu_versions=$(uv run --script "$sourcepath"/build_tools/supported_ubuntu_versions.py)
+
+# Write the specfile
+[ -e "$workpath"/fish.spec ] && { echo "Cowardly refusing to overwite an existing fish.spec" >&2;
+exit 1; }
+rpmversion=$(echo "$version" |sed -e 's/-/+/' -e 's/-/./g')
+sed -e "s/@version@/$version/g" -e "s/@rpmversion@/$rpmversion/g" \
+    < "$sourcepath"/fish.spec.in > "$workpath"/fish.spec
+
+# Make the symlinks for Debian
+ln -s "$source_tarball" "$workpath"/fish_"$version".orig.tar.xz
+ln -s "$vendor_tarball" "$workpath"/fish_"$version".orig-cargo-vendor.tar.xz
+
+# Set up the Debian source tree
+cd "$sourcepath"
+mkdir cargo-vendor
+tar -C cargo-vendor -x -f "$vendor_tarball"
+cp -r contrib/debian debian
+
+# The vendor tarball contains a new .cargo/config.toml, which has the
+# vendoring overrides appended to it. dpkg-source will add this as a
+# patch using the flags in debian/
+cp cargo-vendor/.cargo/config.toml .cargo/config.toml
+
+# Update the Debian changelog
+# The release scripts do this for release builds - skip if it has already been done
+if head -n1 debian/changelog | grep --invert-match --quiet --fixed-strings "$version"; then
+    debchange --newversion "$version-1" --distribution unstable "Snapshot build"
+fi
+
+# Builds the "plain" Debian package
+# debuild runs lintian, which takes ten minutes to run over the vendor directories
+# just use dpkg-buildpackage directly
+dpkg-buildpackage --build=source -d
+
+# Build the Ubuntu packages
+# deb-reversion does not work on source packages, so do the whole thing ourselves
+for series in $ubuntu_versions; do
+    sed -i -e "1 s/$version-1)/$version-1~$series)/" -e "1 s/unstable/$series/" debian/changelog
+    dpkg-buildpackage --build=source -d
+    sed -i -e "1 s/$version-1~$series)/$version-1)/" -e "1 s/$series/unstable/" debian/changelog
+done
+
+}

build_tools/make_tarball.sh

@@ -15,9 +15,8 @@ tmpdir=$(mktemp -d)
 manifest=$tmpdir/Cargo.toml
 lockfile=$tmpdir/Cargo.lock
 
-sed "s/^version = \".*\"\$/version = \"$VERSION\"/g" Cargo.toml \
- >"$manifest"
-awk -v version=$VERSION '
+sed "s/^version = \".*\"\$/version = \"$VERSION\"/g" Cargo.toml >"$manifest"
+awk -v version="$VERSION" '
     /^name = "fish"$/ { ok=1 }
     ok == 1 && /^version = ".*"$/ {
         ok = 2;

build_tools/make_vendor_tarball.sh

@@ -8,20 +8,6 @@
 # Exit on error
 set -e
 
-# We need GNU tar as that supports the --mtime and --transform options
-TAR=notfound
-for try in tar gtar gnutar; do
-    if $try -Pcf /dev/null --mtime now /dev/null >/dev/null 2>&1; then
-        TAR=$try
-        break
-    fi
-done
-
-if [ "$TAR" = "notfound" ]; then
-    echo 'No suitable tar (supporting --mtime) found as tar/gtar/gnutar in PATH'
-    exit 1
-fi
-
 # Get the current directory, which we'll use for telling Cargo where to find the sources
 wd="$PWD"
 

build_tools/osx_package_scripts/add-shell

@@ -10,8 +10,8 @@ fi
 
 scriptname=$(basename "$0")
 if [ "$(id -u)" -ne 0 ]; then
-     echo "${scriptname} must be run as root"
-     exit 1
+    echo "${scriptname} must be run as root"
+    exit 1
 fi
 
 file=/etc/shells

build_tools/osx_package_scripts/preinstall

@@ -2,6 +2,6 @@
 
 echo "Removing any previous installation"
 pkgutil --pkg-info "${INSTALL_PKG_SESSION_ID}" && pkgutil --only-files --files "${INSTALL_PKG_SESSION_ID}" | while read -r installed
-     do rm -v "${DSTVOLUME}${installed}"
+    do rm -v "${DSTVOLUME}${installed}"
 done
 echo "... removed"

build_tools/release-notes.sh

@@ -47,8 +47,8 @@ if test -z "$CI" || [ "$(git -C "$workspace_root" tag | wc -l)" -gt 1 ]; then {
             num_authors=$(wc -l <"$relnotes_tmp/committers-now")
             num_new_authors=$(wc -l <"$relnotes_tmp/committers-new")
             printf %s \
-                "This release comprises $num_commits commits since $previous_version," \
-                " contributed by $num_authors authors, $num_new_authors of which are new committers."
+                "This release brings $num_commits new commits since $previous_version," \
+                " contributed by $num_authors authors, $num_new_authors of which are new faces."
             echo
             echo
         )
@@ -86,10 +86,13 @@ if test -z "$CI" || [ "$(git -C "$workspace_root" tag | wc -l)" -gt 1 ]; then {
     echo
     echo 'Download links:'
     echo 'To download the source code for fish, we suggest the file named ``fish-'"$version"'.tar.xz``.'
+    # shellcheck disable=2016
     echo 'The file downloaded from ``Source code (tar.gz)`` will not build correctly.'
+    # shellcheck disable=2016
     echo 'A GPG signature using `this key <'"${FISH_GPG_PUBLIC_KEY_URL:-???}"'>`__ is available as ``fish-'"$version"'.tar.xz.asc``.'
     echo
     echo 'The files called ``fish-'"$version"'-linux-*.tar.xz`` contain'
+    # shellcheck disable=2016
     echo '`standalone fish binaries <https://github.com/fish-shell/fish-shell/?tab=readme-ov-file#building-fish-with-cargo>`__'
     echo 'for any Linux with the given CPU architecture.'
 } >"$relnotes_tmp/fake-workspace"/CHANGELOG.rst

build_tools/release.sh

@@ -72,7 +72,7 @@ integration_branch=$(
         --format='%(refname:strip=2)'
 )
 [ -n "$integration_branch" ] ||
-    git merge-base --is-ancestor $remote/master HEAD
+    git merge-base --is-ancestor "$remote"/master HEAD
 
 sed -n 1p CHANGELOG.rst | grep -q '^fish .*(released .*)$'
 sed -n 2p CHANGELOG.rst | grep -q '^===*$'
@@ -113,9 +113,9 @@ CreateCommit "Release $version"
 # Tags must be full objects, not lightweight tags, for
 # git_version-gen.sh to work.
 git -c "user.signingKey=$committer" \
-    tag --sign --message="Release $version" $version
+    tag --sign --message="Release $version" "$version"
 
-git push $remote $version
+git push "$remote" "$version"
 
 TIMEOUT=
 gh() {
@@ -173,6 +173,7 @@ actual_tag_oid=$(git ls-remote "$remote" |
 (
     cd "$tmpdir/local-tarball/fish-$version"
     uv --no-managed-python venv
+    # shellcheck disable=1091
     . .venv/bin/activate
     cmake -GNinja -DCMAKE_BUILD_TYPE=Debug .
     ninja doc
@@ -180,14 +181,17 @@ actual_tag_oid=$(git ls-remote "$remote" |
 CopyDocs() {
     rm -rf "$fish_site/site/docs/$1"
     cp -r "$tmpdir/local-tarball/fish-$version/cargo/fish-docs/html" "$fish_site/site/docs/$1"
-    git -C $fish_site add "site/docs/$1"
+    git -C "$fish_site" add "site/docs/$1"
 }
 minor_version=${version%.*}
 CopyDocs "$minor_version"
 latest_release=$(
     releases=$(git tag | grep '^[0-9]*\.[0-9]*\.[0-9]*.*' |
-        sed $(: "De-prioritize release candidates (1.2.3-rc0)") \
-        's/-/~/g' | LC_ALL=C sort --version-sort)
+        sed '
+            # De-prioritize release candidates (1.2.3-rc0)
+            s/-/~/g
+        ' | LC_ALL=C sort --version-sort
+    )
     printf %s\\n "$releases" | tail -1
 )
 if [ "$version" = "$latest_release" ]; then
@@ -251,6 +255,28 @@ do
     sleep 20
 done
 
+milestone_version="$(
+    if echo "$version" | grep -q '\.0$'; then
+        echo "$minor_version"
+    else
+        echo "$version"
+    fi
+)"
+milestone_number() {
+    gh_api_repo milestones?state=open |
+        jq --arg name "fish $1" '
+            .[] | select(.title == $name) | .number
+        '
+}
+gh_api_repo milestones/"$(milestone_number "$milestone_version")" \
+    --method PATCH --raw-field state=closed
+next_minor_version=$(echo "$minor_version" |
+    awk -F. '{ printf "%s.%s", $1, $2+1 }')
+if [ -z "$(milestone_number "$next_minor_version")" ]; then
+    gh_api_repo milestones --method POST \
+        --raw-field title="fish $next_minor_version"
+fi
+
 (
     cd "$fish_site"
     make new-release
@@ -272,7 +298,7 @@ done
 )
 
 if [ -n "$integration_branch" ]; then {
-    git push $remote "$version^{commit}":refs/heads/$integration_branch
+    git push "$remote" "$version^{commit}:refs/heads/$integration_branch"
 } else {
     changelog=$(cat - CHANGELOG.rst <<EOF
 fish ?.?.? (released ???)
@@ -283,32 +309,9 @@ EOF
     printf %s\\n "$changelog" >CHANGELOG.rst
     git add CHANGELOG.rst
     CreateCommit "start new cycle"
-    git push $remote HEAD:master
+    git push "$remote" HEAD:master
 } fi
 
-milestone_version="$(
-    if echo "$version" | grep -q '\.0$'; then
-        echo "$minor_version"
-    else
-        echo "$version"
-    fi
-)"
-milestone_number() {
-    gh_api_repo milestones?state=open |
-        jq --arg name "fish $1" '
-            .[] | select(.title == $name) | .number
-        '
-}
-gh_api_repo milestones/"$(milestone_number "$milestone_version")" \
-    --method PATCH --raw-field state=closed
-
-next_minor_version=$(echo "$minor_version" |
-    awk -F. '{ printf "%s.%s", $1, $2+1 }')
-if [ -z "$(milestone_number "$next_minor_version")" ]; then
-    gh_api_repo milestones --method POST \
-        --raw-field title="fish $next_minor_version"
-fi
-
 exit
 
 }

build_tools/style.fish

@@ -1,123 +0,0 @@
-#!/usr/bin/env fish
-#
-# This runs Python files, fish scripts (*.fish), and Rust files
-# through their respective code formatting programs.
-#
-# `--all`: Format all eligible files instead of the ones specified as arguments.
-# `--check`: Instead of reformatting, fail if a file is not formatted correctly.
-# `--force`: Proceed without asking if uncommitted changes are detected.
-#            Only relevant if `--all` is specified but `--check` is not specified.
-
-set -l fish_files
-set -l python_files
-set -l rust_files
-set -l all no
-
-argparse all check force -- $argv
-or exit $status
-
-if set -l -q _flag_all
-    set all yes
-    if set -q argv[1]
-        echo "Unexpected arguments: '$argv'"
-        exit 1
-    end
-end
-
-set -l workspace_root (realpath (status dirname)/..)
-
-if test $all = yes
-    if not set -l -q _flag_force; and not set -l -q _flag_check
-        # Potential for false positives: Not all fish files are formatted, see the `fish_files`
-        # definition below.
-        set -l relevant_uncommitted_changes (git status --porcelain --short --untracked-files=all | sed -e 's/^ *[^ ]* *//' | grep -E '.*\.(fish|py|rs)$')
-        if set -q relevant_uncommitted_changes[1]
-            for changed_file in $relevant_uncommitted_changes
-                echo $changed_file
-            end
-            echo
-            echo 'You have uncommitted changes (listed above). Are you sure you want to restyle?'
-            read -P 'y/N? ' -n1 -l ans
-            if not string match -qi y -- $ans
-                exit 1
-            end
-        end
-    end
-    set fish_files $workspace_root/{benchmarks,build_tools,etc,share}/**.fish
-    set python_files $workspace_root
-else
-    # Format the files specified as arguments.
-    set -l files $argv
-    set fish_files (string match -r '^.*\.fish$' -- $files)
-    set python_files (string match -r '^.*\.py$' -- $files)
-    set rust_files (string match -r '^.*\.rs$' -- $files)
-end
-
-set -l red (set_color red)
-set -l green (set_color green)
-set -l yellow (set_color yellow)
-set -l normal (set_color normal)
-
-function die -V red -V normal
-    echo $red$argv[1]$normal
-    exit 1
-end
-
-if set -q fish_files[1]
-    if not type -q fish_indent
-        echo
-        echo $yellow'Could not find `fish_indent` in `$PATH`.'$normal
-        exit 127
-    end
-    echo === Running "$green"fish_indent"$normal"
-    if set -l -q _flag_check
-        fish_indent --check -- $fish_files
-        or die "Fish files are not formatted correctly."
-    else
-        fish_indent -w -- $fish_files
-    end
-end
-
-if set -q python_files[1]
-    if not type -q ruff
-        echo
-        echo $yellow'Please install `ruff` to style python'$normal
-        exit 127
-    end
-    echo === Running "$green"ruff format"$normal"
-    if set -l -q _flag_check
-        ruff format --check $python_files
-        or die "Python files are not formatted correctly."
-    else
-        ruff format $python_files
-    end
-end
-
-if test $all = yes; or set -q rust_files[1]
-    if not cargo fmt --version >/dev/null
-        echo
-        echo $yellow'Please install "rustfmt" to style Rust, e.g. via:'
-        echo "rustup component add rustfmt"$normal
-        exit 127
-    end
-
-    set -l edition_spec string match -r '^edition\s*=.*'
-    test "$($edition_spec <Cargo.toml)" = "$($edition_spec <.rustfmt.toml)"
-    or die "Cargo.toml and .rustfmt.toml use different editions"
-
-    echo === Running "$green"rustfmt"$normal"
-    if set -l -q _flag_check
-        if test $all = yes
-            cargo fmt --all --check
-        else
-            rustfmt --check --files-with-diff $rust_files
-        end
-        or die "Rust files are not formatted correctly."
-    else
-        if test $all = yes
-            cargo fmt --all
-        else
-            rustfmt $rust_files
-        end
-    end
-end

build_tools/update-dependencies.sh

@@ -3,7 +3,6 @@
 set -ex
 
 command -v curl
-command -v gcloud
 command -v jq
 command -v rustup
 command -v updatecli
@@ -19,21 +18,24 @@ update_gh_action() {
     [ -n "$version" ]
     tag_oid=$(git ls-remote "https://github.com/$repo.git" "refs/tags/$version" | cut -f1)
     [ -n "$tag_oid" ]
-    find .github/workflows -name '*.yml' -type f -exec \
-        sed -i "s|uses: $repo@\S\+\( \+#.*\)\?|\
-uses: $repo@$tag_oid # $version, build_tools/update-dependencies.sh|g" {} +
+    workflow_files=$(find .github/workflows -name '*.yml' -type f)
+    # shellcheck disable=2086
+    grep "\buses: $repo@\S\+\( \+#.*\)\?" $workflow_files
+    # shellcheck disable=2086
+    sed -i "s|\buses: $repo@\S\+\( \+#.*\)\?|\
+uses: $repo@$tag_oid # $version, build_tools/update-dependencies.sh|g" \
+        $workflow_files
 }
+
 update_gh_action actions/checkout
+update_gh_action actions/download-artifact
 update_gh_action actions/github-script
 update_gh_action actions/upload-artifact
-update_gh_action actions/download-artifact
-update_gh_action docker/login-action
-update_gh_action docker/build-push-action
-update_gh_action docker/metadata-action
-update_gh_action EmbarkStudios/cargo-deny-action
 update_gh_action dessant/lock-threads
-update_gh_action softprops/action-gh-release
+update_gh_action EmbarkStudios/cargo-deny-action
 update_gh_action msys2/setup-msys2
+update_gh_action softprops/action-gh-release
+update_gh_action vmactions/freebsd-vm
 
 updatecli "${@:-apply}"
 
@@ -42,17 +44,18 @@ uv lock --upgrade --exclude-newer="$(date --date='7 days ago' --iso-8601)"
 
 from_gh() {
     repo=$1
-    path=$2
-    destination=$3
-    contents=$(curl -fsS https://raw.githubusercontent.com/"${repo}"/refs/heads/master/"${path}")
-    printf '%s\n' >"$destination" "$contents"
+    branch=$2
+    path=$3
+    destination=$4
+    contents=$(curl -fsS https://raw.githubusercontent.com/"${repo}"/refs/heads/"${branch}"/"${path}")
+    printf '%s\n' "$contents" >"$destination"
 }
 
-from_gh ridiculousfish/widecharwidth widechar_width.rs crates/widecharwidth/src/widechar_width.rs
-from_gh ridiculousfish/littlecheck littlecheck/littlecheck.py tests/littlecheck.py
-from_gh catppuccin/fish 'themes/Catppuccin Frappe.theme' share/themes/catppuccin-frappe.theme
-from_gh catppuccin/fish 'themes/Catppuccin Macchiato.theme' share/themes/catppuccin-macchiato.theme
-from_gh catppuccin/fish 'themes/Catppuccin Mocha.theme' share/themes/catppuccin-mocha.theme
+from_gh ridiculousfish/widecharwidth master widechar_width.rs crates/widecharwidth/src/widechar_width.rs
+from_gh ridiculousfish/littlecheck master littlecheck/littlecheck.py tests/littlecheck.py
+from_gh catppuccin/fish main themes/catppuccin-frappe.theme share/themes/catppuccin-frappe.theme
+from_gh catppuccin/fish main themes/catppuccin-macchiato.theme share/themes/catppuccin-macchiato.theme
+from_gh catppuccin/fish main themes/catppuccin-mocha.theme share/themes/catppuccin-mocha.theme
 
 # Update Cargo.lock
 cargo update

build_tools/update_translations.fish

@@ -1,156 +0,0 @@
-#!/usr/bin/env fish
-
-# Updates the files used for gettext translations.
-# By default, the whole xgettext + msgmerge pipeline runs,
-# which extracts the messages from the source files into $template_file,
-# and updates the PO files for each language from that.
-#
-# Use cases:
-# For developers:
-#   - Run with no args to update all PO files after making changes to Rust/fish sources.
-# For translators:
-#   - Specify the language you want to work on as an argument, which must be a file in the
-#     localization/po/ directory. You can specify a language which does not have translations
-#     yet by specifying the name of a file which does not yet exist.
-#     Make sure to follow the naming convention.
-# For testing:
-#   - Specify `--dry-run` to see if any updates to the PO files would by applied by this script.
-#     If this flag is specified, the script will exit with an error if there are outstanding
-#     changes, and will display the diff. Do not specify other flags if `--dry-run` is specified.
-#
-# Specify `--use-existing-template=DIR` to prevent running cargo for extracting an up-to-date
-# version of the localized strings. This flag is intended for testing setups which make it
-# inconvenient to run cargo here, but run it in an earlier step to ensure up-to-date values.
-# This argument is passed on to the `fish_xgettext.fish` script and has no other uses.
-# `DIR` must be the path to a gettext template file generated from our compilation process.
-# It can be obtained by running:
-#   set -l DIR (mktemp -d)
-#   FISH_GETTEXT_EXTRACTION_DIR=$DIR cargo check --features=gettext-extract
-
-# The sort utility is locale-sensitive.
-# Ensure that sorting output is consistent by setting LC_ALL here.
-set -gx LC_ALL C.UTF-8
-
-set -l build_tools (status dirname)
-set -l po_dir $build_tools/../localization/po
-
-set -l extract
-
-argparse dry-run use-existing-template= -- $argv
-or exit $status
-
-if test -z $argv[1]
-    # Update everything if not specified otherwise.
-    set -g po_files $po_dir/*.po
-else
-    set -l po_dir_id (stat --format='%d:%i' -- $po_dir)
-    for arg in $argv
-        set -l arg_dir_id (stat --format='%d:%i' -- (dirname $arg) 2>/dev/null)
-        if test $po_dir_id != "$arg_dir_id"
-            echo "Argument $arg is not a file in the directory $(realpath $po_dir)."
-            echo "Non-option arguments must specify paths to files in this directory."
-            echo ""
-            echo "If you want to add a new language to the translations not the following:"
-            echo "The filename must identify a language, with a two letter ISO 639-1 language code of the target language (e.g. 'pt' for Portuguese), and use the file extension '.po'."
-            echo "Optionally, you can specify a regional variant (e.g. 'pt_BR')."
-            echo "So valid filenames are of the shape 'll.po' or 'll_CC.po'."
-            exit 1
-        end
-        if not basename $arg | grep -qE '^[a-z]{2,3}(_[A-Z]{2})?\.po$'
-            echo "Filename does not match the expected format ('ll.po' or 'll_CC.po')."
-            exit 1
-        end
-    end
-    set -g po_files $argv
-end
-
-set -g template_file (mktemp)
-# Protect from externally set $tmpdir leaking into this script.
-set -g tmpdir
-
-function cleanup_exit
-    set -l exit_status $status
-
-    rm $template_file
-
-    if set -g --query tmpdir[1]
-        rm -r $tmpdir
-    end
-
-    exit $exit_status
-end
-
-if set -l --query extract
-    set -l xgettext_args
-    if set -l --query _flag_use_existing_template
-        set xgettext_args --use-existing-template=$_flag_use_existing_template
-    end
-    $build_tools/fish_xgettext.fish $xgettext_args >$template_file
-    or cleanup_exit
-end
-
-if set -l --query _flag_dry_run
-    # On a dry run, we do not modify localization/po/ but write to a temporary directory instead
-    # and check if there is a difference between localization/po/ and the tmpdir after re-generating
-    # the PO files.
-    set -g tmpdir (mktemp -d)
-
-    # Ensure tmpdir has the same initial state as the po dir.
-    cp -r $po_dir/* $tmpdir
-end
-
-# This is used to identify lines which should be set here via $header_lines.
-# Make sure that this prefix does not appear elsewhere in the file and only contains characters
-# without special meaning in a sed pattern.
-set -g header_prefix "# fish-note-sections: "
-
-function print_header
-    set -l header_lines \
-        "Translations are divided into sections, each starting with a fish-section-* pseudo-message." \
-        "The first few sections are more important." \
-        "Ignore the tier3 sections unless you have a lot of time."
-    for line in $header_lines
-        printf '%s%s\n' $header_prefix $line
-    end
-end
-
-function merge_po_files --argument-names template_file po_file
-    msgmerge --no-wrap --update --no-fuzzy-matching --backup=none --quiet \
-        $po_file $template_file
-    or cleanup_exit
-    set -l new_po_file (mktemp) # TODO Remove on failure.
-    # Remove obsolete messages instead of keeping them as #~ entries.
-    and msgattrib --no-wrap --no-obsolete -o $new_po_file $po_file
-    or cleanup_exit
-
-    begin
-        print_header
-        # Paste PO file without old header lines.
-        sed '/^'$header_prefix'/d' $new_po_file
-    end >$po_file
-    rm $new_po_file
-end
-
-for po_file in $po_files
-    if set --query tmpdir[1]
-        set po_file $tmpdir/(basename $po_file)
-    end
-    if test -e $po_file
-        merge_po_files $template_file $po_file
-    else
-        begin
-            print_header
-            cat $template_file
-        end >$po_file
-    end
-end
-
-if set -g --query tmpdir[1]
-    diff -ur $po_dir $tmpdir
-    or begin
-        echo ERROR: translations in localization/po/ are stale. Try running build_tools/update_translations.fish
-        cleanup_exit
-    end
-end
-
-cleanup_exit

build_tools/version-available-in-debian.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 set -euo pipefail
 
@@ -11,6 +11,6 @@ codename=$(
 curl -fsS https://sources.debian.org/api/src/"${package}"/ |
     jq -r --arg codename "${codename}" '
         .versions[] | select(.suites[] == $codename) | .version' |
-    sed 's/^\([0-9]\+\.[0-9]\+\).*/\1/' |
+    sed -E 's/^([0-9]+\.[0-9]+).*/\1/' |
     sort --version-sort |
     tail -1

clippy.toml

@@ -0,0 +1,6 @@
+allow-print-in-tests = true
+disallowed-methods = [
+    # Not allowed to use libc::setlocale() directly, need to use the wrappers
+    # in src/locale.rs which takes a lock for the duration of the call.
+    "libc::setlocale",
+]

cmake/Docs.cmake

@@ -51,9 +51,10 @@ endif()
 add_feature_info(Documentation WITH_DOCS "user manual and documentation")
 
 if(WITH_DOCS)
-    add_custom_target(doc ALL
-                      DEPENDS sphinx-docs sphinx-manpages)
+    add_custom_target(doc ALL DEPENDS sphinx-docs sphinx-manpages)
     # Group docs targets into a DocsTargets folder
-    set_property(TARGET doc sphinx-docs sphinx-manpages
-                 PROPERTY FOLDER cmake/DocTargets)
+    set_property(
+        TARGET doc sphinx-docs sphinx-manpages
+        PROPERTY FOLDER cmake/DocTargets
+    )
 endif()

cmake/FindRust.cmake

@@ -22,17 +22,17 @@ foreach(_VAR ${_Rust_USER_VARS})
 endforeach()
 
 if (NOT DEFINED Rust_CARGO_CACHED)
-  find_program(Rust_CARGO_CACHED cargo PATHS "$ENV{HOME}/.cargo/bin")
+    find_program(Rust_CARGO_CACHED cargo PATHS "$ENV{HOME}/.cargo/bin")
 endif()
 
 if (NOT EXISTS "${Rust_CARGO_CACHED}")
-  message(FATAL_ERROR "The cargo executable ${Rust_CARGO_CACHED} was not found. "
-    "Consider setting `Rust_CARGO_CACHED` to the absolute path of `cargo`."
-  )
+    message(FATAL_ERROR "The cargo executable ${Rust_CARGO_CACHED} was not found. "
+        "Consider setting `Rust_CARGO_CACHED` to the absolute path of `cargo`."
+    )
 endif()
 
 if (NOT DEFINED Rust_COMPILER_CACHED)
-  find_program(Rust_COMPILER_CACHED rustc PATHS "$ENV{HOME}/.cargo/bin")
+    find_program(Rust_COMPILER_CACHED rustc PATHS "$ENV{HOME}/.cargo/bin")
 endif()
 
 
@@ -45,31 +45,31 @@ endif()
 # Figure out the target by just using the host target.
 # If you want to cross-compile, you'll have to set Rust_CARGO_TARGET
 if(NOT Rust_CARGO_TARGET_CACHED)
-  execute_process(
-    COMMAND "${Rust_COMPILER_CACHED}" --version --verbose
-    OUTPUT_VARIABLE _RUSTC_VERSION_RAW
-    RESULT_VARIABLE _RUSTC_VERSION_RESULT
-  )
-
-  if(NOT ( "${_RUSTC_VERSION_RESULT}" EQUAL "0" ))
-    message(FATAL_ERROR "Failed to get rustc version.\n"
-      "${Rust_COMPILER} --version failed with error: `${_RUSTC_VERSION_RESULT}`")
-  endif()
-
-  if (_RUSTC_VERSION_RAW MATCHES "host: ([a-zA-Z0-9_\\-]*)\n")
-    set(Rust_DEFAULT_HOST_TARGET "${CMAKE_MATCH_1}")
-  else()
-    message(FATAL_ERROR
-      "Failed to parse rustc host target. `rustc --version --verbose` evaluated to:\n${_RUSTC_VERSION_RAW}"
+    execute_process(
+        COMMAND "${Rust_COMPILER_CACHED}" --version --verbose
+        OUTPUT_VARIABLE _RUSTC_VERSION_RAW
+        RESULT_VARIABLE _RUSTC_VERSION_RESULT
     )
-  endif()
 
-  if(CMAKE_CROSSCOMPILING)
-    message(FATAL_ERROR "CMake is in cross-compiling mode."
-      "Manually set `Rust_CARGO_TARGET`."
-    )
-  endif()
-  set(Rust_CARGO_TARGET_CACHED "${Rust_DEFAULT_HOST_TARGET}" CACHE STRING "Target triple")
+    if(NOT ( "${_RUSTC_VERSION_RESULT}" EQUAL "0" ))
+        message(FATAL_ERROR "Failed to get rustc version.\n"
+            "${Rust_COMPILER} --version failed with error: `${_RUSTC_VERSION_RESULT}`")
+    endif()
+
+    if (_RUSTC_VERSION_RAW MATCHES "host: ([a-zA-Z0-9_\\-]*)\n")
+        set(Rust_DEFAULT_HOST_TARGET "${CMAKE_MATCH_1}")
+    else()
+        message(FATAL_ERROR
+            "Failed to parse rustc host target. `rustc --version --verbose` evaluated to:\n${_RUSTC_VERSION_RAW}"
+        )
+    endif()
+
+    if(CMAKE_CROSSCOMPILING)
+        message(FATAL_ERROR "CMake is in cross-compiling mode."
+            "Manually set `Rust_CARGO_TARGET`."
+        )
+    endif()
+    set(Rust_CARGO_TARGET_CACHED "${Rust_DEFAULT_HOST_TARGET}" CACHE STRING "Target triple")
 endif()
 
 # Set the input variables as non-cache variables so that the variables are available after

cmake/Install.cmake

@@ -14,32 +14,36 @@ set(rel_completionsdir "fish/vendor_completions.d")
 set(rel_functionsdir "fish/vendor_functions.d")
 set(rel_confdir "fish/vendor_conf.d")
 
-set(extra_completionsdir
-    "${datadir}/${rel_completionsdir}"
-    CACHE STRING "Path for extra completions")
+set(
+    extra_completionsdir "${datadir}/${rel_completionsdir}"
+    CACHE STRING "Path for extra completions"
+)
 
-set(extra_functionsdir
-    "${datadir}/${rel_functionsdir}"
-    CACHE STRING "Path for extra functions")
+set(
+    extra_functionsdir "${datadir}/${rel_functionsdir}"
+    CACHE STRING "Path for extra functions"
+)
 
-set(extra_confdir
-    "${datadir}/${rel_confdir}"
-    CACHE STRING "Path for extra configuration")
+set(
+    extra_confdir "${datadir}/${rel_confdir}"
+    CACHE STRING "Path for extra configuration"
+)
 
 
 # These are the man pages that go in system manpath; all manpages go in the fish-specific manpath.
-set(MANUALS ${SPHINX_OUTPUT_DIR}/man/man1/fish.1
-            ${SPHINX_OUTPUT_DIR}/man/man1/fish_indent.1
-            ${SPHINX_OUTPUT_DIR}/man/man1/fish_key_reader.1
-            ${SPHINX_OUTPUT_DIR}/man/man1/fish-doc.1
-            ${SPHINX_OUTPUT_DIR}/man/man1/fish-tutorial.1
-            ${SPHINX_OUTPUT_DIR}/man/man1/fish-language.1
-            ${SPHINX_OUTPUT_DIR}/man/man1/fish-interactive.1
-            ${SPHINX_OUTPUT_DIR}/man/man1/fish-terminal-compatibility.1
-            ${SPHINX_OUTPUT_DIR}/man/man1/fish-completions.1
-            ${SPHINX_OUTPUT_DIR}/man/man1/fish-prompt-tutorial.1
-            ${SPHINX_OUTPUT_DIR}/man/man1/fish-for-bash-users.1
-            ${SPHINX_OUTPUT_DIR}/man/man1/fish-faq.1
+set(MANUALS
+    ${SPHINX_OUTPUT_DIR}/man/man1/fish.1
+    ${SPHINX_OUTPUT_DIR}/man/man1/fish_indent.1
+    ${SPHINX_OUTPUT_DIR}/man/man1/fish_key_reader.1
+    ${SPHINX_OUTPUT_DIR}/man/man1/fish-doc.1
+    ${SPHINX_OUTPUT_DIR}/man/man1/fish-tutorial.1
+    ${SPHINX_OUTPUT_DIR}/man/man1/fish-language.1
+    ${SPHINX_OUTPUT_DIR}/man/man1/fish-interactive.1
+    ${SPHINX_OUTPUT_DIR}/man/man1/fish-terminal-compatibility.1
+    ${SPHINX_OUTPUT_DIR}/man/man1/fish-completions.1
+    ${SPHINX_OUTPUT_DIR}/man/man1/fish-prompt-tutorial.1
+    ${SPHINX_OUTPUT_DIR}/man/man1/fish-for-bash-users.1
+    ${SPHINX_OUTPUT_DIR}/man/man1/fish-faq.1
 )
 
 # Determine which man page we don't want to install.
@@ -48,40 +52,49 @@ set(MANUALS ${SPHINX_OUTPUT_DIR}/man/man1/fish.1
 # On other operating systems, don't install a realpath man page, as they almost all have a realpath
 # command, while macOS does not.
 if(${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
-  set(CONDEMNED_PAGE "open.1")
+    set(CONDEMNED_PAGE "open.1")
 else()
-  set(CONDEMNED_PAGE "realpath.1")
+    set(CONDEMNED_PAGE "realpath.1")
 endif()
 
 # Define a function to help us create directories.
 function(FISH_CREATE_DIRS)
-  foreach(dir ${ARGV})
-    install(DIRECTORY DESTINATION ${dir})
-  endforeach(dir)
+    foreach(dir ${ARGV})
+        install(DIRECTORY DESTINATION ${dir})
+    endforeach(dir)
 endfunction(FISH_CREATE_DIRS)
 
 function(FISH_TRY_CREATE_DIRS)
-  foreach(dir ${ARGV})
-    if(NOT IS_ABSOLUTE ${dir})
-      set(abs_dir "\$ENV{DESTDIR}\${CMAKE_INSTALL_PREFIX}/${dir}")
-    else()
-      set(abs_dir "\$ENV{DESTDIR}${dir}")
-    endif()
-    install(SCRIPT CODE "EXECUTE_PROCESS(COMMAND mkdir -p ${abs_dir} OUTPUT_QUIET ERROR_QUIET)
-                         execute_process(COMMAND chmod 755 ${abs_dir} OUTPUT_QUIET ERROR_QUIET)
-                        ")
-  endforeach()
+    foreach(dir ${ARGV})
+        if(NOT IS_ABSOLUTE ${dir})
+            set(abs_dir "\$ENV{DESTDIR}\${CMAKE_INSTALL_PREFIX}/${dir}")
+        else()
+            set(abs_dir "\$ENV{DESTDIR}${dir}")
+        endif()
+        install(SCRIPT CODE "
+            EXECUTE_PROCESS(COMMAND mkdir -p ${abs_dir} OUTPUT_QUIET ERROR_QUIET)
+            execute_process(COMMAND chmod 755 ${abs_dir} OUTPUT_QUIET ERROR_QUIET)
+        ")
+    endforeach()
 endfunction(FISH_TRY_CREATE_DIRS)
 
-install(PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/fish
-        PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ
-                    GROUP_EXECUTE WORLD_READ WORLD_EXECUTE
-        DESTINATION ${bindir})
+install(
+    PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/fish
+    PERMISSIONS
+        OWNER_READ
+        OWNER_WRITE
+        OWNER_EXECUTE
+        GROUP_READ
+        GROUP_EXECUTE
+        WORLD_READ
+        WORLD_EXECUTE
+    DESTINATION ${bindir}
+)
 
 if(NOT IS_ABSOLUTE ${bindir})
-  set(abs_bindir "\$ENV{DESTDIR}\${CMAKE_INSTALL_PREFIX}/${bindir}")
+    set(abs_bindir "\$ENV{DESTDIR}\${CMAKE_INSTALL_PREFIX}/${bindir}")
 else()
-  set(abs_bindir "\$ENV{DESTDIR}${bindir}")
+    set(abs_bindir "\$ENV{DESTDIR}${bindir}")
 endif()
 install(CODE "file(CREATE_LINK ${abs_bindir}/fish ${abs_bindir}/fish_indent)")
 install(CODE "file(CREATE_LINK ${abs_bindir}/fish ${abs_bindir}/fish_key_reader)")
@@ -90,87 +103,68 @@ fish_create_dirs(${sysconfdir}/fish/conf.d ${sysconfdir}/fish/completions
     ${sysconfdir}/fish/functions)
 install(FILES etc/config.fish DESTINATION ${sysconfdir}/fish/)
 
-fish_create_dirs(${rel_datadir}/fish ${rel_datadir}/fish/completions
-                 ${rel_datadir}/fish/functions
-                 ${rel_datadir}/fish/man/man1 ${rel_datadir}/fish/tools
-                 ${rel_datadir}/fish/tools/web_config
-                 ${rel_datadir}/fish/tools/web_config/js
-                 ${rel_datadir}/fish/prompts
-                 ${rel_datadir}/fish/themes
-                 )
+fish_create_dirs(
+    ${rel_datadir}/fish
+    ${rel_datadir}/fish/man/man1
+)
 
+# This file is embedded in the executable by rust-embed and never read from the filesystem
 configure_file(share/__fish_build_paths.fish.in share/__fish_build_paths.fish)
-install(FILES share/config.fish
-              ${CMAKE_CURRENT_BINARY_DIR}/share/__fish_build_paths.fish
-        DESTINATION ${rel_datadir}/fish)
 
 # Create only the vendor directories inside the prefix (#5029 / #6508)
-fish_create_dirs(${rel_datadir}/fish/vendor_completions.d ${rel_datadir}/fish/vendor_functions.d
-    ${rel_datadir}/fish/vendor_conf.d)
+fish_create_dirs(
+    ${rel_datadir}/fish/vendor_completions.d
+    ${rel_datadir}/fish/vendor_functions.d
+    ${rel_datadir}/fish/vendor_conf.d
+)
 
 fish_try_create_dirs(${rel_datadir}/pkgconfig)
 configure_file(fish.pc.in fish.pc.noversion @ONLY)
 
-add_custom_command(OUTPUT fish.pc
+add_custom_command(
+    OUTPUT fish.pc
     COMMAND sed '/Version/d' fish.pc.noversion > fish.pc
     COMMAND printf "Version: " >> fish.pc
     COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/build_tools/git_version_gen.sh >> fish.pc
     WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
-    DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/fish.pc.noversion)
+    DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/fish.pc.noversion
+)
 
 add_custom_target(build_fish_pc ALL DEPENDS fish.pc)
 
-install(FILES ${CMAKE_CURRENT_BINARY_DIR}/fish.pc
-        DESTINATION ${rel_datadir}/pkgconfig)
-
-install(DIRECTORY share/completions/
-        DESTINATION ${rel_datadir}/fish/completions
-        FILES_MATCHING PATTERN "*.fish")
-
-install(DIRECTORY share/functions/
-        DESTINATION ${rel_datadir}/fish/functions
-        FILES_MATCHING PATTERN "*.fish")
-
-install(DIRECTORY share/prompts/
-        DESTINATION ${rel_datadir}/fish/prompts
-        FILES_MATCHING PATTERN "*.fish")
-
-install(DIRECTORY share/themes/
-        DESTINATION ${rel_datadir}/fish/themes
-        FILES_MATCHING PATTERN "*.theme")
+install(
+    FILES ${CMAKE_CURRENT_BINARY_DIR}/fish.pc
+    DESTINATION ${rel_datadir}/pkgconfig
+)
 
 # CONDEMNED_PAGE is managed by the conditional above
 # Building the man pages is optional: if sphinx isn't installed, they're not built
-install(DIRECTORY ${SPHINX_OUTPUT_DIR}/man/man1/
-        DESTINATION ${rel_datadir}/fish/man/man1
-        FILES_MATCHING
-        PATTERN "*.1"
-        PATTERN ${CONDEMNED_PAGE} EXCLUDE)
-
-install(PROGRAMS share/tools/create_manpage_completions.py
-        DESTINATION ${rel_datadir}/fish/tools/)
-
-install(DIRECTORY share/tools/web_config
-        DESTINATION ${rel_datadir}/fish/tools/
-        FILES_MATCHING
-        PATTERN "*.png"
-        PATTERN "*.css"
-        PATTERN "*.html"
-        PATTERN "*.py"
-        PATTERN "*.js")
+install(
+    DIRECTORY ${SPHINX_OUTPUT_DIR}/man/man1/
+    DESTINATION ${rel_datadir}/fish/man/man1
+    FILES_MATCHING
+    PATTERN "*.1"
+    PATTERN ${CONDEMNED_PAGE} EXCLUDE
+)
 
 # Building the man pages is optional: if Sphinx isn't installed, they're not built
 install(FILES ${MANUALS} DESTINATION ${mandir}/man1/ OPTIONAL)
-install(DIRECTORY ${SPHINX_OUTPUT_DIR}/html/ # Trailing slash is important!
-        DESTINATION ${docdir} OPTIONAL)
+install(
+    DIRECTORY ${SPHINX_OUTPUT_DIR}/html/ # Trailing slash is important!
+    DESTINATION ${docdir} OPTIONAL
+)
 install(FILES CHANGELOG.rst DESTINATION ${docdir})
 
 # Group install targets into a InstallTargets folder
-set_property(TARGET build_fish_pc
-             PROPERTY FOLDER cmake/InstallTargets)
+set_property(
+    TARGET build_fish_pc
+    PROPERTY FOLDER cmake/InstallTargets
+)
 
 # Make a target build_root that installs into the buildroot directory, for testing.
 set(BUILDROOT_DIR ${CMAKE_CURRENT_BINARY_DIR}/buildroot)
-add_custom_target(build_root
-                  COMMAND DESTDIR=${BUILDROOT_DIR} ${CMAKE_COMMAND}
-                          --build ${CMAKE_CURRENT_BINARY_DIR} --target install)
+add_custom_target(
+    build_root
+    COMMAND DESTDIR=${BUILDROOT_DIR} ${CMAKE_COMMAND}
+        --build ${CMAKE_CURRENT_BINARY_DIR} --target install
+)

cmake/PCRE2.cmake

@@ -1,9 +1,10 @@
 set(FISH_USE_SYSTEM_PCRE2 ON CACHE BOOL
-  "Try to use PCRE2 from the system, instead of the pcre2-sys version")
+    "Try to use PCRE2 from the system, instead of the pcre2-sys version"
+)
 
 if(FISH_USE_SYSTEM_PCRE2)
-  message(STATUS "Trying to use PCRE2 from the system")
+    message(STATUS "Trying to use PCRE2 from the system")
 else()
-  message(STATUS "Forcing static build of PCRE2")
-  set(FISH_PCRE2_BUILDFLAG "PCRE2_SYS_STATIC=1")
+    message(STATUS "Forcing static build of PCRE2")
+    set(FISH_PCRE2_BUILDFLAG "PCRE2_SYS_STATIC=1")
 endif(FISH_USE_SYSTEM_PCRE2)

cmake/Tests.cmake

@@ -1,27 +1,27 @@
 FILE(GLOB FISH_CHECKS CONFIGURE_DEPENDS ${CMAKE_SOURCE_DIR}/tests/checks/*.fish)
 foreach(CHECK ${FISH_CHECKS})
-  get_filename_component(CHECK_NAME ${CHECK} NAME)
-  add_custom_target(
-    test_${CHECK_NAME}
-    COMMAND ${CMAKE_SOURCE_DIR}/tests/test_driver.py ${CMAKE_CURRENT_BINARY_DIR}
-                checks/${CHECK_NAME}
-    WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/tests
-    DEPENDS fish fish_indent fish_key_reader
-    USES_TERMINAL
-  )
+    get_filename_component(CHECK_NAME ${CHECK} NAME)
+    add_custom_target(
+        test_${CHECK_NAME}
+        COMMAND ${CMAKE_SOURCE_DIR}/tests/test_driver.py ${CMAKE_CURRENT_BINARY_DIR}
+            checks/${CHECK_NAME}
+        WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/tests
+        DEPENDS fish fish_indent fish_key_reader
+        USES_TERMINAL
+    )
 endforeach(CHECK)
 
 FILE(GLOB PEXPECTS CONFIGURE_DEPENDS ${CMAKE_SOURCE_DIR}/tests/pexpects/*.py)
 foreach(PEXPECT ${PEXPECTS})
-  get_filename_component(PEXPECT ${PEXPECT} NAME)
-  add_custom_target(
-    test_${PEXPECT}
-    COMMAND ${CMAKE_SOURCE_DIR}/tests/test_driver.py ${CMAKE_CURRENT_BINARY_DIR}
-                pexpects/${PEXPECT}
-    WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/tests
-    DEPENDS fish fish_indent fish_key_reader
-    USES_TERMINAL
-  )
+    get_filename_component(PEXPECT ${PEXPECT} NAME)
+    add_custom_target(
+        test_${PEXPECT}
+        COMMAND ${CMAKE_SOURCE_DIR}/tests/test_driver.py ${CMAKE_CURRENT_BINARY_DIR}
+            pexpects/${PEXPECT}
+        WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/tests
+        DEPENDS fish fish_indent fish_key_reader
+        USES_TERMINAL
+    )
 endforeach(PEXPECT)
 
 # Rust stuff.
@@ -30,14 +30,20 @@ if(DEFINED ASAN)
     # Rust w/ -Zsanitizer=address requires explicitly specifying the --target triple or else linker
     # errors pertaining to asan symbols will ensue.
     if(NOT DEFINED Rust_CARGO_TARGET)
-        message(FATAL_ERROR "ASAN requires defining the CMake variable Rust_CARGO_TARGET to the
-            intended target triple")
+        message(
+            FATAL_ERROR
+            "ASAN requires defining the CMake variable Rust_CARGO_TARGET to the
+            intended target triple"
+        )
     endif()
 endif()
 if(DEFINED TSAN)
     if(NOT DEFINED Rust_CARGO_TARGET)
-        message(FATAL_ERROR "TSAN requires defining the CMake variable Rust_CARGO_TARGET to the
-            intended target triple")
+        message(
+            FATAL_ERROR
+            "TSAN requires defining the CMake variable Rust_CARGO_TARGET to the
+            intended target triple"
+        )
     endif()
 endif()
 
@@ -53,10 +59,10 @@ endif()
 
 # The top-level test target is "fish_run_tests".
 add_custom_target(fish_run_tests
-  # TODO: This should be replaced with a unified solution, possibly build_tools/check.sh.
-  COMMAND ${CMAKE_SOURCE_DIR}/tests/test_driver.py ${max_concurrency_flag} ${CMAKE_CURRENT_BINARY_DIR}
-  COMMAND env ${VARS_FOR_CARGO}
-            ${Rust_CARGO}
+    # TODO: This should be replaced with a unified solution, possibly build_tools/check.sh.
+    COMMAND ${CMAKE_SOURCE_DIR}/tests/test_driver.py ${max_concurrency_flag} ${CMAKE_CURRENT_BINARY_DIR}
+    COMMAND
+        env ${VARS_FOR_CARGO} ${Rust_CARGO}
             test
             --no-default-features
             --features=${FISH_CARGO_FEATURES}
@@ -64,7 +70,7 @@ add_custom_target(fish_run_tests
             --workspace
             --target-dir ${rust_target_dir}
             ${cargo_test_flags}
-  WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
-  DEPENDS fish fish_indent fish_key_reader
-  USES_TERMINAL
+    WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
+    DEPENDS fish fish_indent fish_key_reader
+    USES_TERMINAL
 )

contrib/debian/changelog

@@ -1,3 +1,27 @@
+fish (4.7.1-1) stable; urgency=medium
+
+  * Release of new version 4.7.1.
+
+  See https://github.com/fish-shell/fish-shell/releases/tag/4.7.1 for details.
+
+ -- Johannes Altmanninger <aclopte@gmail.com>  Fri, 08 May 2026 00:02:14 +0800
+
+fish (4.7.0-1) stable; urgency=medium
+
+  * Release of new version 4.7.0.
+
+  See https://github.com/fish-shell/fish-shell/releases/tag/4.7.0 for details.
+
+ -- Johannes Altmanninger <aclopte@gmail.com>  Tue, 05 May 2026 15:24:27 +0800
+
+fish (4.6.0-1) stable; urgency=medium
+
+  * Release of new version 4.6.0.
+
+  See https://github.com/fish-shell/fish-shell/releases/tag/4.6.0 for details.
+
+ -- Johannes Altmanninger <aclopte@gmail.com>  Sat, 28 Mar 2026 12:56:37 +0800
+
 fish (4.5.0-1) stable; urgency=medium
 
   * Release of new version 4.5.0.

contrib/debian/control

@@ -10,10 +10,12 @@ Build-Depends: debhelper-compat (= 13),
  gettext,
  libpcre2-dev,
  rustc (>= 1.85) | rustc-web (>= 1.85) | rustc-1.85,
+# pkg-config is needed for the pcre2 crate to find the pcre2 system library
+ pkgconf | pkg-config,
  python3-sphinx,
 # Test dependencies
  locales-all,
- man-db,
+ man-db | man,
  python3
 # 4.6.2 is Debian 12/Ubuntu Noble 24.04; Ubuntu Jammy is 4.6.0.1
 Standards-Version: 4.6.2
@@ -26,8 +28,8 @@ Architecture: any
 # for col and lock
 Depends: bsdextrautils,
  file,
-# for man
- man-db,
+# for showing built-in help pages
+ man-db | man,
 # for kill
  procps,
  python3 (>=3.5),

contrib/shell.nix

@@ -0,0 +1,33 @@
+# Environment containing all dependencies needed for
+# - building fish,
+# - building documentation,
+# - running all tests,
+# - formatting and checking lints.
+#
+# enter interactive bash shell:
+#   nix-shell contrib/shell.nix
+#
+# using system nixpkgs (otherwise fetches pinned version):
+#   nix-shell contrib/shell.nix --arg pkgs 'import <nixpkgs> {}'
+#
+# run single command:
+#   nix-shell contrib/shell --run "cargo xtask check"
+{ pkgs ? (import (builtins.fetchTarball {
+  url = "https://github.com/NixOS/nixpkgs/archive/nixos-25.11.tar.gz";
+  sha256 = "1ia5kjykm9xmrpwbzhbaf4cpwi3yaxr7shl6amj8dajvgbyh2yh4";
+}) { }), ... }:
+pkgs.mkShell {
+  buildInputs = [
+    (pkgs.python3.withPackages (pyPkgs: [ pyPkgs.pexpect ]))
+    pkgs.cargo
+    pkgs.clippy
+    pkgs.cmake
+    pkgs.gettext
+    pkgs.pcre2
+    pkgs.procps # tests use pgrep/pkill
+    pkgs.ruff
+    pkgs.rustc
+    pkgs.rustfmt
+    pkgs.sphinx
+  ];
+}

crates/build-helper/Cargo.toml

@@ -1,8 +1,8 @@
 [package]
 name = "fish-build-helper"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 

crates/build-helper/src/lib.rs

@@ -38,6 +38,24 @@ pub fn fish_doc_dir() -> Cow<'static, Path> {
     cargo_target_dir().join("fish-docs").into()
 }
 
+fn l10n_dir() -> Cow<'static, Path> {
+    workspace_root().join("localization").into()
+}
+
+pub fn ftl_dir() -> Cow<'static, Path> {
+    l10n_dir().join("fluent").into()
+}
+
+pub static DEFAULT_LANGUAGE: &str = "en";
+
+pub fn default_ftl_file() -> Cow<'static, Path> {
+    ftl_dir().join(format!("{DEFAULT_LANGUAGE}.ftl")).into()
+}
+
+pub fn po_dir() -> Cow<'static, Path> {
+    l10n_dir().join("po").into()
+}
+
 // TODO Move this to rsconf
 pub fn rebuild_if_path_changed<P: AsRef<Path>>(path: P) {
     rsconf::rebuild_if_path_changed(path.as_ref().to_str().unwrap());
@@ -90,14 +108,14 @@ pub fn target_os_is_cygwin() -> bool {
 
 #[macro_export]
 macro_rules! as_os_strs {
-    [ $( $x:expr, )* ] => {
+    [ $( $x:expr ),* $(,)? ] => {
         {
             use std::ffi::OsStr;
             fn as_os_str<S: AsRef<OsStr> + ?Sized>(s: &S) -> &OsStr {
                 s.as_ref()
             }
-            &[
-                $( as_os_str($x), )*
+            [
+                $( as_os_str($x) ),*
             ]
         }
     }

crates/build-man-pages/Cargo.toml

@@ -1,8 +1,8 @@
 [package]
 name = "fish-build-man-pages"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 

crates/color/Cargo.toml

@@ -1,8 +1,8 @@
 [package]
 name = "fish-color"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 

crates/common/Cargo.toml

@@ -1,16 +1,21 @@
 [package]
 name = "fish-common"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 
 [dependencies]
 bitflags.workspace = true
+fish-feature-flags.workspace = true
 fish-widestring.workspace = true
 libc.workspace = true
 nix.workspace = true
 
+[build-dependencies]
+fish-build-helper.workspace = true
+rsconf.workspace = true
+
 [lints]
 workspace = true

crates/common/build.rs

@@ -0,0 +1,5 @@
+use fish_build_helper::target_os_is_apple;
+
+fn main() {
+    rsconf::declare_cfg("apple", target_os_is_apple());
+}

crates/fallback/Cargo.toml

@@ -1,8 +1,8 @@
 [package]
 name = "fish-fallback"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 

crates/fallback/src/lib.rs

@@ -8,12 +8,12 @@
 use std::cmp;
 use std::sync::{
     LazyLock,
-    atomic::{AtomicIsize, Ordering},
+    atomic::{AtomicUsize, Ordering},
 };
 
 /// Width of ambiguous East Asian characters and, as of TR11, all private-use characters.
 /// 1 is the typical default, but we accept any non-negative override via `$fish_ambiguous_width`.
-pub static FISH_AMBIGUOUS_WIDTH: AtomicIsize = AtomicIsize::new(1);
+pub static FISH_AMBIGUOUS_WIDTH: AtomicUsize = AtomicUsize::new(1);
 
 /// Width of emoji characters.
 ///
@@ -25,68 +25,33 @@
 /// Valid values are 1, and 2. 1 is the typical emoji width used in Unicode 8 while some newer
 /// terminals use a width of 2 since Unicode 9.
 // For some reason, this is declared here and exposed here, but is set in `env_dispatch`.
-pub static FISH_EMOJI_WIDTH: AtomicIsize = AtomicIsize::new(1);
+pub static FISH_EMOJI_WIDTH: AtomicUsize = AtomicUsize::new(2);
 
 static WC_LOOKUP_TABLE: LazyLock<WcLookupTable> = LazyLock::new(WcLookupTable::new);
 
-/// A safe wrapper around the system `wcwidth()` function
-#[cfg(not(cygwin))]
-pub fn wcwidth(c: char) -> isize {
-    unsafe extern "C" {
-        pub unsafe fn wcwidth(c: libc::wchar_t) -> libc::c_int;
-    }
-
-    const {
-        assert!(size_of::<libc::wchar_t>() >= size_of::<char>());
-    }
-
-    let width = unsafe { wcwidth(c as libc::wchar_t) };
-    isize::try_from(width).unwrap()
-}
-
-// Big hack to use our versions of wcswidth where we know them to be broken, which is
-// EVERYWHERE (https://github.com/fish-shell/fish-shell/issues/2199)
-pub fn fish_wcwidth(c: char) -> isize {
-    // The system version of wcwidth should accurately reflect the ability to represent characters
-    // in the console session, but knows nothing about the capabilities of other terminal emulators
-    // or ttys. Use it from the start only if we are logged in to the physical console.
-    #[cfg(not(cygwin))]
-    if fish_common::is_console_session() {
-        return wcwidth(c);
-    }
-
+pub fn fish_wcwidth(c: char) -> Option<usize> {
     // Check for VS16 which selects emoji presentation. This "promotes" a character like U+2764
     // (width 1) to an emoji (probably width 2). So treat it as width 1 so the sums work. See #2652.
     // VS15 selects text presentation.
     let variation_selector_16 = '\u{FE0F}';
     let variation_selector_15 = '\u{FE0E}';
     if c == variation_selector_16 {
-        return 1;
+        return Some(1);
     } else if c == variation_selector_15 {
-        return 0;
+        return Some(0);
     }
 
     // Check for Emoji_Modifier property. Only the Fitzpatrick modifiers have this, in range
     // 1F3FB..1F3FF. This is a hack because such an emoji appearing on its own would be drawn as
     // width 2, but that's unlikely to be useful. See #8275.
     if ('\u{1F3FB}'..='\u{1F3FF}').contains(&c) {
-        return 0;
+        return Some(0);
     }
 
     let width = WC_LOOKUP_TABLE.classify(c);
-    match width {
-        WcWidth::NonCharacter | WcWidth::NonPrint | WcWidth::Combining | WcWidth::Unassigned => {
-            #[cfg(not(cygwin))]
-            {
-                // Fall back to system wcwidth in this case.
-                wcwidth(c)
-            }
-            #[cfg(cygwin)]
-            {
-                // No system wcwidth for UTF-32 on cygwin.
-                0
-            }
-        }
+    Some(match width {
+        WcWidth::NonPrint => return None,
+        WcWidth::NonCharacter | WcWidth::Combining | WcWidth::Unassigned => 0,
         WcWidth::Ambiguous | WcWidth::PrivateUse => {
             // TR11: "All private-use characters are by default classified as Ambiguous".
             FISH_AMBIGUOUS_WIDTH.load(Ordering::Relaxed)
@@ -94,26 +59,25 @@ pub fn fish_wcwidth(c: char) -> isize {
         WcWidth::One => 1,
         WcWidth::Two => 2,
         WcWidth::WidenedIn9 => FISH_EMOJI_WIDTH.load(Ordering::Relaxed),
-    }
+    })
 }
 
-/// fish's internal versions of wcwidth and wcswidth, which can use an internal implementation if
-/// the system one is busted.
-pub fn fish_wcswidth(s: &wstr) -> isize {
-    // ascii fast path; empty iterator returns true for .all()
-    if s.chars().all(|c| c.is_ascii() && !c.is_ascii_control()) {
-        return s.len() as isize;
+pub fn fish_wcswidth(s: &wstr) -> Option<usize> {
+    fish_wcswidth_canonicalizing(s, std::convert::identity)
+}
+
+pub fn fish_wcswidth_canonicalizing(s: &wstr, canonicalize: fn(char) -> char) -> Option<usize> {
+    let chars = s.chars().map(canonicalize);
+    // ascii fast path
+    if chars.clone().all(|c| c.is_ascii() && !c.is_ascii_control()) {
+        return Some(s.len());
     }
 
     let mut result = 0;
-    for c in s.chars() {
-        let w = fish_wcwidth(c);
-        if w < 0 {
-            return -1;
-        }
-        result += w;
+    for c in chars {
+        result += fish_wcwidth(c)?;
     }
-    result
+    Some(result)
 }
 
 pub fn wcscasecmp(lhs: &wstr, rhs: &wstr) -> cmp::Ordering {

crates/feature-flags/Cargo.toml

@@ -0,0 +1,13 @@
+[package]
+name = "fish-feature-flags"
+version = "0.0.0"
+edition.workspace = true
+rust-version.workspace = true
+repository.workspace = true
+license.workspace = true
+
+[dependencies]
+fish-widestring.workspace = true
+
+[lints]
+workspace = true

crates/feature-flags/src/lib.rs

@@ -1,15 +1,14 @@
 //! Flags to enable upcoming features
 
-use crate::prelude::*;
-use std::sync::atomic::AtomicBool;
-use std::sync::atomic::Ordering;
-
-#[cfg(test)]
-use std::cell::RefCell;
+use fish_widestring::{L, WExt as _, wstr};
+use std::{
+    cell::RefCell,
+    sync::atomic::{AtomicBool, Ordering},
+};
 
 /// The list of flags.
 #[repr(u8)]
-#[derive(Clone, Copy)]
+#[derive(Clone, Copy, PartialEq, Eq)]
 pub enum FeatureFlag {
     /// Whether ^ is supported for stderr redirection.
     StderrNoCaret,
@@ -63,10 +62,10 @@ pub struct FeatureMetadata {
     pub description: &'static wstr,
 
     /// Default flag value.
-    pub default_value: bool,
+    default_value: bool,
 
     /// Whether the value can still be changed or not.
-    pub read_only: bool,
+    read_only: bool,
 }
 
 /// The metadata, indexed by flag.
@@ -156,31 +155,26 @@ pub struct FeatureMetadata {
 ];
 
 thread_local!(
-    #[cfg(test)]
-    static LOCAL_FEATURES: RefCell<Option<Features>> = const { RefCell::new(None) };
+    static LOCAL_OVERRIDE_STACK: RefCell<Vec<(FeatureFlag, bool)>> =
+        const { RefCell::new(Vec::new()) };
 );
 
 /// The singleton shared feature set.
 static FEATURES: Features = Features::new();
 
 /// Perform a feature test on the global set of features.
-pub fn test(flag: FeatureFlag) -> bool {
-    #[cfg(test)]
-    {
-        LOCAL_FEATURES.with(|fc| fc.borrow().as_ref().unwrap_or(&FEATURES).test(flag))
+pub fn feature_test(flag: FeatureFlag) -> bool {
+    if let Some(value) = LOCAL_OVERRIDE_STACK.with(|stack| {
+        for &(overridden_feature, value) in stack.borrow().iter().rev() {
+            if flag == overridden_feature {
+                return Some(value);
+            }
+        }
+        None
+    }) {
+        return value;
     }
-    #[cfg(not(test))]
-    {
-        FEATURES.test(flag)
-    }
-}
-
-pub use test as feature_test;
-
-/// Set a flag.
-#[cfg(test)]
-pub fn set(flag: FeatureFlag, value: bool) {
-    LOCAL_FEATURES.with(|fc| fc.borrow().as_ref().unwrap_or(&FEATURES).set(flag, value));
+    FEATURES.test(flag)
 }
 
 /// Parses a comma-separated feature-flag string, updating ourselves with the values.
@@ -188,20 +182,7 @@ pub fn set(flag: FeatureFlag, value: bool) {
 /// The special group name "all" may be used for those who like to live on the edge.
 /// Unknown features are silently ignored.
 pub fn set_from_string<'a>(str: impl Into<&'a wstr>) {
-    let wstr: &wstr = str.into();
-    #[cfg(test)]
-    {
-        LOCAL_FEATURES.with(|fc| {
-            fc.borrow()
-                .as_ref()
-                .unwrap_or(&FEATURES)
-                .set_from_string(wstr);
-        });
-    }
-    #[cfg(not(test))]
-    {
-        FEATURES.set_from_string(wstr);
-    }
+    FEATURES.set_from_string(str.into());
 }
 
 impl Features {
@@ -237,19 +218,14 @@ fn set(&self, flag: FeatureFlag, value: bool) {
     }
 
     fn set_from_string(&self, str: &wstr) {
-        let whitespace = L!("\t\n\0x0B\0x0C\r ").as_char_slice();
-        for entry in str.as_char_slice().split(|c| *c == ',') {
+        for entry in str.split(',') {
+            let entry = entry.trim();
             if entry.is_empty() {
                 continue;
             }
 
-            // Trim leading and trailing whitespace
-            let entry = &entry[entry.iter().take_while(|c| whitespace.contains(c)).count()..];
-            let entry =
-                &entry[..entry.len() - entry.iter().take_while(|c| whitespace.contains(c)).count()];
-
             // A "no-" prefix inverts the sense.
-            let (name, value) = match entry.strip_prefix(L!("no-").as_char_slice()) {
+            let (name, value) = match entry.strip_prefix("no-") {
                 Some(suffix) => (suffix, false),
                 None => (entry, true),
             };
@@ -275,28 +251,20 @@ fn set_from_string(&self, str: &wstr) {
     }
 }
 
-#[cfg(test)]
-pub fn scoped_test(flag: FeatureFlag, value: bool, test_fn: impl FnOnce()) {
-    LOCAL_FEATURES.with(|fc| {
-        assert!(
-            fc.borrow().is_none(),
-            "scoped_test() does not support nesting"
-        );
-
-        let f = Features::new();
-        f.set(flag, value);
-        *fc.borrow_mut() = Some(f);
-
+/// Run code with a feature overridden.
+/// This should only be used in tests.
+pub fn with_overridden_feature(flag: FeatureFlag, value: bool, test_fn: impl FnOnce()) {
+    LOCAL_OVERRIDE_STACK.with(|stack| {
+        stack.borrow_mut().push((flag, value));
         test_fn();
-
-        *fc.borrow_mut() = None;
+        stack.borrow_mut().pop();
     });
 }
 
 #[cfg(test)]
 mod tests {
-    use super::{FeatureFlag, Features, METADATA, scoped_test, set, test};
-    use crate::prelude::*;
+    use super::{FeatureFlag, Features, METADATA, feature_test, with_overridden_feature};
+    use fish_widestring::L;
 
     #[test]
     fn test_feature_flags() {
@@ -322,25 +290,19 @@ fn test_feature_flags() {
     }
 
     #[test]
-    fn test_scoped() {
-        scoped_test(FeatureFlag::QuestionMarkNoGlob, true, || {
-            assert!(test(FeatureFlag::QuestionMarkNoGlob));
+    fn test_overridden_feature() {
+        with_overridden_feature(FeatureFlag::QuestionMarkNoGlob, true, || {
+            assert!(feature_test(FeatureFlag::QuestionMarkNoGlob));
         });
 
-        set(FeatureFlag::QuestionMarkNoGlob, true);
-
-        scoped_test(FeatureFlag::QuestionMarkNoGlob, false, || {
-            assert!(!test(FeatureFlag::QuestionMarkNoGlob));
+        with_overridden_feature(FeatureFlag::QuestionMarkNoGlob, false, || {
+            assert!(!feature_test(FeatureFlag::QuestionMarkNoGlob));
         });
 
-        set(FeatureFlag::QuestionMarkNoGlob, false);
-    }
-
-    #[test]
-    #[should_panic]
-    fn test_nested_scopes_not_supported() {
-        scoped_test(FeatureFlag::QuestionMarkNoGlob, true, || {
-            scoped_test(FeatureFlag::QuestionMarkNoGlob, false, || {});
+        with_overridden_feature(FeatureFlag::QuestionMarkNoGlob, false, || {
+            with_overridden_feature(FeatureFlag::QuestionMarkNoGlob, true, || {
+                assert!(feature_test(FeatureFlag::QuestionMarkNoGlob));
+            });
         });
     }
 }

crates/fluent-extraction/Cargo.toml

@@ -0,0 +1,24 @@
+[package]
+name = "fish-fluent-extraction"
+edition.workspace = true
+rust-version.workspace = true
+version = "0.0.0"
+repository.workspace = true
+license.workspace = true
+description = "proc-macro for extracting IDs for fluent translation"
+
+[lib]
+proc-macro = true
+
+[dependencies]
+fish-tempfile.workspace = true
+fluent-ftl-tools.workspace = true
+fluent-syntax.workspace = true
+proc-macro2.workspace = true
+syn.workspace = true
+
+[build-dependencies]
+rsconf.workspace = true
+
+[lints]
+workspace = true

crates/fluent-extraction/build.rs

@@ -0,0 +1,3 @@
+fn main() {
+    rsconf::rebuild_if_env_changed("FISH_FLUENT_EXTRACTION_DIR");
+}

crates/fluent-extraction/src/lib.rs

@@ -0,0 +1,105 @@
+extern crate proc_macro;
+use fluent_ftl_tools::{
+    HasEntries as _, format_resource, parse_str_as_syntax_resource, serialize_resource,
+};
+use proc_macro::TokenStream;
+use std::{
+    collections::HashSet,
+    ffi::{OsStr, OsString},
+    io::Write as _,
+    path::PathBuf,
+};
+use syn::{
+    Ident, LitStr, Token,
+    parse::{Parse, ParseStream},
+    parse_macro_input,
+};
+
+struct LocalizeDefinition {
+    message_id: String,
+    message_definition: String,
+    variables: Vec<String>,
+}
+
+impl Parse for LocalizeDefinition {
+    fn parse(input: ParseStream) -> syn::Result<Self> {
+        let message_id = input.parse::<LitStr>()?.value();
+        input.parse::<Token![=]>()?;
+        let message_definition = input.parse::<LitStr>()?.value();
+        if !input.is_empty() {
+            input.parse::<Token![,]>()?;
+        }
+
+        fn parse_key(input: ParseStream) -> syn::Result<String> {
+            let key = input.parse::<Ident>()?;
+            Ok(format!("{key}"))
+        }
+        let variables = Vec::from_iter(input.parse_terminated(parse_key, Token![,])?);
+        Ok(Self {
+            message_id,
+            message_definition,
+            variables,
+        })
+    }
+}
+
+fn check(localize_definition: &LocalizeDefinition) -> String {
+    let message = format!(
+        "{} = {}",
+        localize_definition.message_id, localize_definition.message_definition
+    );
+    let resource = parse_str_as_syntax_resource(&message)
+        .unwrap_or_else(|err| panic!("Failed to parse Fluent message\n{message}\n\n{err}"));
+    let resource_entries = &resource.body;
+    assert_eq!(
+        resource_entries.len(),
+        1,
+        "Expected exactly one Fluent entry specified via macro."
+    );
+    assert!(
+        matches!(resource_entries[0], fluent_syntax::ast::Entry::Message(_)),
+        "Expected definition of Fluent message, but got {:?}",
+        resource_entries[0]
+    );
+    let formatted_resource = format_resource(resource.clone())
+        .unwrap_or_else(|err| panic!("Resource is not formatted correctly:\n{err}"));
+    let formatted_resource_string = serialize_resource(&formatted_resource);
+    let formatted_resource_string_trimmed = formatted_resource_string.trim();
+    assert!(
+        message == formatted_resource_string_trimmed,
+        "Message is not formatted correctly.\nActual:\n{message}\nExpected:\n{formatted_resource_string_trimmed}"
+    );
+    let mut expected_variables = HashSet::new();
+    for variable in &localize_definition.variables {
+        assert!(
+            expected_variables.insert(variable.as_str()),
+            "Variable {variable} is used as a key more than once."
+        );
+    }
+    resource.check_if_expected_variables_match_message(&expected_variables, &localize_definition.message_id).unwrap_or_else(|err| {
+        panic!("Variables used in macro key-value pairs do not match variables used in Fluent message:\n{err}");
+    });
+    message
+}
+
+fn extract(message: &str, dir_path: &OsStr) {
+    let dir = PathBuf::from(dir_path);
+    let (path, result) = fish_tempfile::create_file_with_retry(|| {
+        dir.join(fish_tempfile::random_filename(OsString::new()))
+    });
+    let mut file = result.unwrap_or_else(|e| {
+        panic!("Failed to create temporary file {path:?}:\n{e}");
+    });
+    file.write_all(message.as_bytes()).unwrap();
+    file.write_all(b"\n").unwrap();
+}
+
+#[proc_macro]
+pub fn fluent_extract(input: TokenStream) -> TokenStream {
+    if let Some(dir_path) = std::env::var_os("FISH_FLUENT_EXTRACTION_DIR") {
+        let localize_definition = parse_macro_input!(input as LocalizeDefinition);
+        let message = check(&localize_definition);
+        extract(&message, &dir_path);
+    }
+    TokenStream::new()
+}

crates/fluent/Cargo.toml

@@ -0,0 +1,30 @@
+[package]
+name = "fish-fluent"
+edition.workspace = true
+rust-version.workspace = true
+version = "0.0.0"
+repository.workspace = true
+license.workspace = true
+
+[dependencies]
+cfg-if.workspace = true
+fish-build-helper.workspace = true
+fish-localization.workspace = true
+fish-fluent-extraction = { workspace = true, optional = true }
+fluent.workspace = true
+rust-embed.workspace = true
+unic-langid.workspace = true
+widestring.workspace = true
+
+[build-dependencies]
+fish-build-helper.workspace = true
+
+[dev-dependencies]
+fluent-ftl-tools.workspace = true
+
+[features]
+fluent-extract = ["dep:fish-fluent-extraction"]
+localize-messages = []
+
+[lints]
+workspace = true

crates/fluent/build.rs

@@ -0,0 +1,4 @@
+fn main() {
+    use fish_build_helper::{ftl_dir, rebuild_if_embedded_path_changed};
+    rebuild_if_embedded_path_changed(ftl_dir());
+}

crates/fluent/src/lib.rs

@@ -0,0 +1,360 @@
+use std::{
+    borrow::Cow,
+    collections::{HashMap, HashSet},
+    fmt::Write as _,
+    sync::{LazyLock, Mutex},
+};
+
+use cfg_if::cfg_if;
+use fish_localization::{
+    DEFAULT_LANGUAGE, Language, LocalizationLanguage, define_localization_language_type,
+};
+use fluent::{
+    FluentArgs, FluentResource, FluentValue, concurrent::FluentBundle, types::FluentNumber,
+};
+use rust_embed::RustEmbed;
+
+#[cfg(feature = "fluent-extract")]
+pub extern crate fish_fluent_extraction;
+
+use unic_langid::LanguageIdentifier;
+
+type Bundle = &'static FluentBundle<FluentResource>;
+
+type NamedBundle = (Language<'static>, Bundle);
+pub type LocalizedMessage = Cow<'static, str>;
+
+static LANGUAGE_BUNDLES: LazyLock<Mutex<HashMap<Language, Bundle>>> =
+    LazyLock::new(|| Mutex::new(HashMap::new()));
+
+static DEFAULT_BUNDLE: LazyLock<NamedBundle> =
+    LazyLock::new(|| (DEFAULT_LANGUAGE, make_bundle(DEFAULT_LANGUAGE)));
+
+static LANGUAGE_BUNDLE_PRECEDENCE: LazyLock<Mutex<Vec<NamedBundle>>> =
+    LazyLock::new(|| Mutex::new(vec![*DEFAULT_BUNDLE]));
+
+cfg_if!(
+    if #[cfg(feature = "localize-messages")] {
+        #[derive(RustEmbed)]
+        #[folder = "../../localization/fluent/"]
+        #[include = "*.ftl"]
+        struct FtlFiles;
+    } else {
+        #[derive(RustEmbed)]
+        #[folder = "../../localization/fluent/"]
+        #[include = "en.ftl"]
+        struct FtlFiles;
+    }
+);
+
+define_localization_language_type! {FluentLocalizationLanguage}
+
+static AVAILABLE_LANGUAGES: LazyLock<HashSet<FluentLocalizationLanguage>> = LazyLock::new(|| {
+    HashSet::from_iter(FtlFiles::iter().map(|language| {
+        let suffix = ".ftl";
+        let language = Language(match language {
+            Cow::Borrowed(language) => language.strip_suffix(suffix).unwrap(),
+            Cow::Owned(mut language) => {
+                assert!(language.ends_with(suffix));
+                language.truncate(language.len() - suffix.len());
+                Box::leak(Box::new(language))
+            }
+        });
+        FluentLocalizationLanguage(language)
+    }))
+});
+
+pub fn get_available_languages() -> &'static HashSet<FluentLocalizationLanguage> {
+    &AVAILABLE_LANGUAGES
+}
+
+fn read_ftl_file(path: &str) -> String {
+    let file = FtlFiles::get(path)
+        .unwrap_or_else(|| panic!("Tried to get FTL file {path} which does not exist."));
+    String::from_utf8(Vec::from(file.data))
+        .unwrap_or_else(|e| panic!("Content of {path} is not valid UTF-8: {e}"))
+}
+
+fn make_bundle(language: Language<'static>) -> Bundle {
+    let langid: LanguageIdentifier = language
+        .parse()
+        .map_err(|e| format!("Failed to parse language identifier {language}: {e}"))
+        .unwrap();
+    let mut bundle = FluentBundle::new_concurrent(vec![langid]);
+    let file_data = read_ftl_file(&format!("{language}.ftl"));
+    // Error handling could use fluent_ftl_tools::display_parse_errors(), but that would require
+    // making the crate a regular dependency.
+    match FluentResource::try_new(file_data) {
+        Ok(res) => {
+            bundle
+                .add_resource(res)
+                .map_err(|e| {
+                    format!(
+                        "Failed to add FTL resources to the bundle for language {language}: {e:?}"
+                    )
+                })
+                .unwrap();
+            // Isolation marks can result in undesirable behavior if the terminal does not support
+            // them properly.
+            // Turn this off for now.
+            // If we add detection for terminal support, we could enable it conditionally.
+            // Without these marks, text order can be incorrect when right-to-left characters are
+            // involved.
+            // https://www.w3.org/International/questions/qa-bidi-unicode-controls
+            // https://github.com/fish-shell/fish-shell/pull/11928#discussion_r2488850606
+            bundle.set_use_isolating(false);
+            // Leak to create static reference.
+            Box::leak(Box::new(bundle))
+        }
+        Err((_resource, errors)) => {
+            let mut error_string = format!("Errors parsing FTL file for {language}:\n");
+            for error in errors {
+                let _ = writeln!(error_string, "{error}");
+            }
+            panic!("{error_string}");
+        }
+    }
+}
+
+/// Set the order in which languages should be tried for localization.
+/// The default language (en) will be added to the end of the list.
+/// If the provided list contains `en`, anything after it will be ignored, since we assume that every
+/// message can be localized in English, so there is no point in specifying further fallback
+/// options.
+/// This function also takes care of lazily loading and parsing data from the embedded FTL files,
+/// so no additional initialization is needed.
+pub fn set_language_precedence(precedence: &[FluentLocalizationLanguage]) {
+    let new_precedence = {
+        let mut bundles = LANGUAGE_BUNDLES.lock().unwrap();
+        // Only take the languages preceding the default language, since it is assumed that everything
+        // is translatable in the default language.
+        let mut new_precedence: Vec<_> = precedence
+            .iter()
+            .take_while(|&lang| Language::from(lang) != DEFAULT_LANGUAGE)
+            .map(|lang| {
+                let language = lang.into();
+                // If a bundle already exists for the language, use it.
+                // Otherwise create a new one and cache it.
+                let bundle = bundles.get(&language).copied().unwrap_or_else(|| {
+                    let bundle = make_bundle(language);
+                    bundles.insert(language, bundle);
+                    bundle
+                });
+                (language, bundle)
+            })
+            .collect();
+        // Add the default language at the end of the precedence list.
+        new_precedence.push(*DEFAULT_BUNDLE);
+        new_precedence
+    };
+    *LANGUAGE_BUNDLE_PRECEDENCE.lock().unwrap() = new_precedence;
+}
+
+pub fn get_language_precedence() -> Vec<Language<'static>> {
+    let language_precedence = LANGUAGE_BUNDLE_PRECEDENCE.lock().unwrap();
+    language_precedence.iter().map(|&(lang, _)| lang).collect()
+}
+
+/// Use the [`localize!`] macro instead of calling this directly.
+/// Panics on errors.
+pub fn format_localized(id: &str, args: &FluentArgs) -> LocalizedMessage {
+    let mut errors = vec![];
+    let bundle_precedence = LANGUAGE_BUNDLE_PRECEDENCE.lock().unwrap();
+    for (_, bundle) in bundle_precedence.iter() {
+        let Some(message) = bundle.get_message(id) else {
+            continue;
+        };
+        let pattern = message.value().expect("Message has no value.");
+        let value = bundle.format_pattern(pattern, Some(args), &mut errors);
+        // NOTE: Unused arguments are not considered errors.
+        if !errors.is_empty() {
+            let mut error_message = format!(
+                "Unexpected formatting errors occurred for message ID '{id}' in language '{}':\n",
+                bundle.locales[0].language
+            );
+            for error in errors {
+                let _ = writeln!(error_message, "{error}");
+            }
+            panic!("{error_message}");
+        }
+        return value;
+    }
+    panic!("Message '{id}' not available in any catalog.")
+}
+
+/// Call this to localize a message with Fluent.
+/// The first argument is a string literal, which defines a Fluent message ID.
+/// It is followed by a mandatory `=` and the English version of the message as a string literal.
+/// The message definition must be valid Fluent syntax. Specifically it must be permissible as the
+/// definition of a Fluent message.
+/// In the simplest case, it will just be a regular string.
+/// If Fluent variables should be used, they need to appear in the message definition, e.g.
+/// `{ $example_variable }`. Then, the variable must also be specified as a key-value pair in the
+/// arguments of the [`localize!`] macro, as demonstrated in the example below.
+/// The key is the Fluent variable name, which also needs to be a syntactically valid Rust
+/// identifier, and the value is whatever the variable should be replaced by when formatting the
+/// localized message.
+/// It is considered an error if the variables specified in the message definition do not match the
+/// variables specified via keys in subsequent arguments to the macro.
+/// The order of key-value pairs can be chosen and modified arbitrarily, and variables may appear
+/// more than once in the message definition.
+///
+/// ```
+/// # use fish_fluent::localize;
+/// # // Note that `localize!` usage in doc-texts is not checked.
+/// let example_message = localize!("test-with-args" = "Two arguments: { $first }, { $second }", first = "foo", second = 42);
+/// assert_eq!(example_message, "Two arguments: foo, 42");
+/// ```
+///
+/// Note that changing the message ID or the message definition has consequences for translations.
+/// If such a change is made, our tooling will automatically detect it and require the developer
+/// making the change to specify what should happen to translations.
+/// See `cargo xtask fluent resolve-outdated`.
+#[macro_export]
+macro_rules! localize {
+    ($id:literal = $message:literal $(, $key:ident = $value:expr)* $(,)?) => {
+        {
+            use $crate::ToFluentValue as _;
+            #[cfg(feature = "fluent-extract")]
+            fish_fluent_extraction::fluent_extract!($id = $message $(, $key)*);
+            let mut args = fluent::FluentArgs::new();
+            $(
+                args.set(stringify!($key), $value.to_fluent_value());
+            )*
+            $crate::format_localized($id, &args)
+        }
+    };
+}
+
+/// Define a function calling [`localize!`] and returning the result.
+/// This macro can be used when multiple locations need access to the same message.
+/// Do not use [`localize!`] with the same message ID more than once.
+/// Instead, define a function which takes the Fluent variables of the message as arguments and
+/// internally calls [`localize!`]. Then, use this function wherever the message is needed.
+/// This macro helps with avoiding some boilerplate. Its first argument is the name of the function
+/// which should be defined. Then, a key-value pair specifying the message ID and English definition
+/// follows, in the same format as for [`localize!`]. The remaining arguments are the names of
+/// Fluent variables which appear in the message definition. These will become the function's
+/// arguments names and they will be used in the internal [`localize!`] call.
+#[macro_export]
+macro_rules! localize_fn {
+    ($vis:vis $fn:ident, $id:literal = $message:literal $(, $key:ident )* $(,)?) => {
+        $vis fn $fn<'a>($($key: impl $crate::ToFluentValue<'a>),*) -> $crate::LocalizedMessage {
+            localize!(
+                $id = $message,
+                $($key = $key),*
+            )
+        }
+    };
+}
+
+/// Trait to account for types which don't have a `Into<FluentValue>` implementation, i.e.
+/// widestrings.
+/// Can be removed once we no longer use such types.
+pub trait ToFluentValue<'a> {
+    fn to_fluent_value(self) -> FluentValue<'a>;
+}
+
+macro_rules! impl_to_fluent_value_wrapper {
+    ($($t:ty),* $(,)?) => {
+        $(
+            impl<'a> ToFluentValue<'a> for $t {
+                fn to_fluent_value(self) -> FluentValue<'a> {
+                    self.into()
+                }
+            }
+        )*
+    };
+}
+impl_to_fluent_value_wrapper! {
+    FluentNumber,
+    String,
+    f32, f64,
+    i8, i16, i32, i64, i128, isize,
+    u8, u16, u32, u64, u128, usize,
+}
+
+macro_rules! impl_to_fluent_value_wrapper_ref {
+    ($($t:ty),* $(,)?) => {
+        $(
+            impl<'a> ToFluentValue<'a> for &'a $t {
+                fn to_fluent_value(self) -> FluentValue<'a> {
+                    self.into()
+                }
+            }
+        )*
+    };
+}
+impl_to_fluent_value_wrapper_ref! {
+    String, str,
+    f32, f64,
+    i8, i16, i32, i64, i128, isize,
+    u8, u16, u32, u64, u128, usize,
+}
+
+impl<'a, T: Into<FluentValue<'a>>> ToFluentValue<'a> for Option<T> {
+    fn to_fluent_value(self) -> FluentValue<'a> {
+        self.into()
+    }
+}
+
+impl<'a> ToFluentValue<'a> for char {
+    fn to_fluent_value(self) -> FluentValue<'a> {
+        self.to_string().into()
+    }
+}
+
+impl<'a> ToFluentValue<'a> for &char {
+    fn to_fluent_value(self) -> FluentValue<'a> {
+        self.to_string().into()
+    }
+}
+
+impl<'a> ToFluentValue<'a> for widestring::Utf32String {
+    fn to_fluent_value(self) -> FluentValue<'a> {
+        self.to_string().into()
+    }
+}
+
+impl<'a> ToFluentValue<'a> for &widestring::Utf32String {
+    fn to_fluent_value(self) -> FluentValue<'a> {
+        self.to_string().into()
+    }
+}
+
+impl<'a> ToFluentValue<'a> for &widestring::Utf32Str {
+    fn to_fluent_value(self) -> FluentValue<'a> {
+        self.to_string().into()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::{DEFAULT_LANGUAGE, FtlFiles};
+    use crate::read_ftl_file;
+    use fluent_ftl_tools::consistency::check_all_resources;
+
+    #[test]
+    fn test_simple_message() {
+        let message_key_value = localize!(
+            "test-with-args" = "Two arguments: { $first }, { $second }",
+            first = "foo",
+            second = 42,
+        );
+        assert_eq!(message_key_value, "Two arguments: foo, 42");
+    }
+
+    #[test]
+    fn check_ftl_files() {
+        let default_resource_name = format!("{DEFAULT_LANGUAGE}.ftl");
+        let default_string = read_ftl_file(&default_resource_name);
+        let other_resources = FtlFiles::iter()
+            .map(|file_path| {
+                let file_string = read_ftl_file(&file_path);
+                (file_path.to_string(), file_string)
+            })
+            .collect::<Vec<_>>();
+        check_all_resources((&default_resource_name, default_string), other_resources)
+            .unwrap_or_else(|e| panic!("FTL resource checks failed:\n{e}"));
+    }
+}

crates/gettext-extraction/Cargo.toml

@@ -1,11 +1,11 @@
 [package]
 name = "fish-gettext-extraction"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
+description = "proc-macro for extracting strings for gettext translation"
 repository.workspace = true
 license.workspace = true
-description = "proc-macro for extracting strings for gettext translation"
 
 [lib]
 proc-macro = true

crates/gettext-maps/Cargo.toml

@@ -1,17 +1,19 @@
 [package]
 name = "fish-gettext-maps"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 
 [dependencies]
+fish-localization.workspace = true
 phf.workspace = true
 
 [build-dependencies]
 fish-build-helper.workspace = true
 fish-gettext-mo-file-parser.workspace = true
+fish-localization.workspace = true
 phf_codegen.workspace = true
 rsconf.workspace = true
 

crates/gettext-maps/build.rs

@@ -5,17 +5,14 @@
 };
 
 use fish_build_helper::env_var;
+use fish_localization::Language;
 
 fn main() {
     let cache_dir =
         PathBuf::from(fish_build_helper::fish_build_dir()).join("fish-localization-map-cache");
     embed_localizations(&cache_dir);
 
-    fish_build_helper::rebuild_if_path_changed(
-        fish_build_helper::workspace_root()
-            .join("localization")
-            .join("po"),
-    );
+    fish_build_helper::rebuild_if_path_changed(fish_build_helper::po_dir());
 }
 
 fn embed_localizations(cache_dir: &Path) {
@@ -25,10 +22,6 @@ fn embed_localizations(cache_dir: &Path) {
         io::{BufWriter, Write as _},
     };
 
-    let po_dir = fish_build_helper::workspace_root()
-        .join("localization")
-        .join("po");
-
     // Ensure that the directory is created, because clippy cannot compile the code if the
     // directory does not exist.
     std::fs::create_dir_all(cache_dir).unwrap();
@@ -56,22 +49,22 @@ fn embed_localizations(cache_dir: &Path) {
         Ok(output) => {
             let has_check_format =
                 String::from_utf8_lossy(&output.stdout).contains("--check-format");
-            for dir_entry_result in po_dir.read_dir().unwrap() {
+            for dir_entry_result in fish_build_helper::po_dir().read_dir().unwrap() {
                 let dir_entry = dir_entry_result.unwrap();
                 let po_file_path = dir_entry.path();
                 if po_file_path.extension() != Some(OsStr::new("po")) {
                     continue;
                 }
-                let lang = po_file_path
+                let language = po_file_path
                     .file_stem()
                     .expect("All entries in the po directory must be regular files.");
-                let language = lang.to_str().unwrap().to_owned();
+                let language = language.to_str().unwrap();
 
                 // Each language gets its own static map for the mapping from message in the source code to
                 // the localized version.
                 let map_name = format!("LANG_MAP_{language}");
 
-                let cached_map_path = cache_dir.join(lang);
+                let cached_map_path = cache_dir.join(language);
 
                 // Include the file containing the map for this language in the main generated file.
                 writeln!(
@@ -82,7 +75,10 @@ fn embed_localizations(cache_dir: &Path) {
                 .unwrap();
                 // Map from the language identifier to the map containing the localizations for this
                 // language.
-                catalogs.entry(language, format!("&{map_name}"));
+                catalogs.entry(
+                    Language(Box::leak(Box::new(language.to_owned()))),
+                    format!("&{map_name}"),
+                );
 
                 if let Ok(metadata) = std::fs::metadata(&cached_map_path) {
                     // Cached map file exists, but might be outdated.
@@ -147,7 +143,7 @@ fn to_raw_str(s: &str) -> String {
                 write!(
                     &mut cached_map_file,
                     "static {}: phf::Map<&'static str, &'static str> = {}",
-                    &map_name,
+                    map_name,
                     single_language_localization_map.build()
                 )
                 .unwrap();
@@ -158,7 +154,8 @@ fn to_raw_str(s: &str) -> String {
 
     write!(
         &mut localization_map_file,
-        "pub static CATALOGS: phf::Map<&str, &phf::Map<&str, &str>> = {}",
+        "use fish_localization::Language;\n\
+         pub static CATALOGS: phf::Map<Language, &phf::Map<&str, &str>> = {}",
         catalogs.build()
     )
     .unwrap();

crates/gettext-mo-file-parser/Cargo.toml

@@ -1,8 +1,8 @@
 [package]
 name = "fish-gettext-mo-file-parser"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 

crates/gettext/Cargo.toml

@@ -1,13 +1,14 @@
 [package]
 name = "fish-gettext"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 
 [dependencies]
 fish-gettext-maps.workspace = true
+fish-localization.workspace = true
 phf.workspace = true
 
 [lints]

crates/gettext/src/lib.rs

@@ -1,12 +1,13 @@
 use fish_gettext_maps::CATALOGS;
+use fish_localization::{Language, LocalizationLanguage, define_localization_language_type};
 use std::{
-    collections::HashMap,
+    collections::HashSet,
     sync::{LazyLock, Mutex},
 };
 
 type Catalog = &'static phf::Map<&'static str, &'static str>;
 
-static LANGUAGE_PRECEDENCE: Mutex<Vec<(&'static str, Catalog)>> = Mutex::new(Vec::new());
+static LANGUAGE_PRECEDENCE: Mutex<Vec<(Language, Catalog)>> = Mutex::new(Vec::new());
 
 pub fn gettext(message_str: &'static str) -> Option<&'static str> {
     let language_precedence = LANGUAGE_PRECEDENCE.lock().unwrap();
@@ -20,21 +21,17 @@ pub fn gettext(message_str: &'static str) -> Option<&'static str> {
     None
 }
 
-#[derive(Clone, Copy)]
-pub struct GettextLocalizationLanguage {
-    language: &'static str,
-}
+define_localization_language_type! {GettextLocalizationLanguage}
 
-static AVAILABLE_LANGUAGES: LazyLock<HashMap<&'static str, GettextLocalizationLanguage>> =
-    LazyLock::new(|| {
-        HashMap::from_iter(
-            CATALOGS
-                .entries()
-                .map(|(&language, _)| (language, GettextLocalizationLanguage { language })),
-        )
-    });
+static AVAILABLE_LANGUAGES: LazyLock<HashSet<GettextLocalizationLanguage>> = LazyLock::new(|| {
+    HashSet::from_iter(
+        CATALOGS
+            .entries()
+            .map(|(&language, _)| GettextLocalizationLanguage(language)),
+    )
+});
 
-pub fn get_available_languages() -> &'static HashMap<&'static str, GettextLocalizationLanguage> {
+pub fn get_available_languages() -> &'static HashSet<GettextLocalizationLanguage> {
     &AVAILABLE_LANGUAGES
 }
 
@@ -43,9 +40,9 @@ pub fn set_language_precedence(new_precedence: &[GettextLocalizationLanguage]) {
         .iter()
         .map(|lang| {
             (
-                lang.language,
+                lang.into(),
                 *CATALOGS
-                    .get(lang.language)
+                    .get(lang.as_ref())
                     .expect("Only languages for which catalogs exist may be passed to gettext."),
             )
         })
@@ -53,7 +50,7 @@ pub fn set_language_precedence(new_precedence: &[GettextLocalizationLanguage]) {
     *LANGUAGE_PRECEDENCE.lock().unwrap() = catalogs;
 }
 
-pub fn get_language_precedence() -> Vec<&'static str> {
+pub fn get_language_precedence() -> Vec<Language<'static>> {
     let language_precedence = LANGUAGE_PRECEDENCE.lock().unwrap();
     language_precedence.iter().map(|&(lang, _)| lang).collect()
 }

crates/localization/Cargo.toml

@@ -0,0 +1,14 @@
+[package]
+name = "fish-localization"
+version = "0.0.0"
+edition.workspace = true
+rust-version.workspace = true
+repository.workspace = true
+license.workspace = true
+
+[dependencies]
+fish-build-helper.workspace = true
+phf_shared.workspace = true
+
+[lints]
+workspace = true

crates/localization/src/lib.rs

@@ -0,0 +1,86 @@
+use std::{borrow::Borrow, hash::Hash};
+
+use phf_shared::{FmtConst, PhfBorrow, PhfHash};
+
+#[derive(Clone, Copy, Eq, Hash, Ord, PartialEq, PartialOrd)]
+pub struct Language<'a>(pub &'a str);
+
+pub const DEFAULT_LANGUAGE: Language = Language(fish_build_helper::DEFAULT_LANGUAGE);
+
+impl<'a> std::ops::Deref for Language<'a> {
+    type Target = &'a str;
+    fn deref(&self) -> &Self::Target {
+        &self.0
+    }
+}
+
+impl<'a> std::fmt::Display for Language<'a> {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        self.0.fmt(f)
+    }
+}
+
+impl AsRef<str> for Language<'_> {
+    fn as_ref(&self) -> &str {
+        self.0
+    }
+}
+
+impl Borrow<str> for Language<'_> {
+    fn borrow(&self) -> &str {
+        self.0
+    }
+}
+
+impl<'a> PhfHash for Language<'a> {
+    fn phf_hash<H: std::hash::Hasher>(&self, state: &mut H) {
+        self.0.phf_hash(state);
+    }
+}
+
+impl<'a> FmtConst for Language<'a> {
+    fn fmt_const(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.write_str("Language(")?;
+        self.0.fmt_const(f)?;
+        f.write_str(")")
+    }
+}
+
+impl<'a> PhfBorrow<Language<'a>> for Language<'a> {
+    fn borrow(&self) -> &Language<'a> {
+        self
+    }
+}
+
+pub trait LocalizationLanguage:
+    AsRef<Language<'static>> + Clone + Copy + Eq + Hash + Ord + PartialEq + PartialOrd + Borrow<str>
+{
+}
+
+#[macro_export]
+macro_rules! define_localization_language_type {
+    ($name:ident) => {
+        #[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
+        pub struct $name(Language<'static>);
+
+        impl LocalizationLanguage for $name {}
+
+        impl AsRef<Language<'static>> for $name {
+            fn as_ref(&self) -> &Language<'static> {
+                &self.0
+            }
+        }
+
+        impl From<&$name> for Language<'static> {
+            fn from(value: &$name) -> Self {
+                value.0
+            }
+        }
+
+        impl std::borrow::Borrow<str> for $name {
+            fn borrow(&self) -> &str {
+                self.0.borrow()
+            }
+        }
+    };
+}

crates/printf/Cargo.toml

@@ -1,10 +1,10 @@
 [package]
 name = "fish-printf"
+version = "0.2.1"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.2.1"
-repository.workspace = true
 description = "printf implementation, based on musl"
+repository.workspace = true
 license = "MIT"
 
 [dependencies]

crates/printf/src/tests.rs

@@ -400,8 +400,10 @@ fn test_char() {
 
 #[test]
 fn test_ptr() {
-    assert_fmt!("%p", core::ptr::null::<u8>() => "0");
-    assert_fmt!("%p", 0xDEADBEEF_usize as *const u8 => "0xdeadbeef");
+    assert_fmt!("%p", core::ptr::null::<()>() => "0");
+
+    let tmp = core::ptr::without_provenance::<()>(0xDEADBEEF);
+    assert_fmt!("%p", tmp => "0xdeadbeef");
 }
 
 #[test]

crates/tempfile/Cargo.toml

@@ -1,13 +1,13 @@
 [package]
 name = "fish-tempfile"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 
 [dependencies]
-nix = { workspace = true, features = ["fs", "feature"] }
+nix = { workspace = true, features = ["feature", "fs"] }
 rand.workspace = true
 
 [lints]

crates/tempfile/src/lib.rs

@@ -7,6 +7,7 @@
 
 use rand::distr::{Alphanumeric, Distribution as _};
 
+#[must_use]
 pub struct TempFile {
     file: File,
     path: PathBuf,
@@ -32,6 +33,7 @@ fn drop(&mut self) {
     }
 }
 
+#[must_use]
 pub struct TempDir {
     path: PathBuf,
 }
@@ -119,7 +121,7 @@ mod tests {
 
     #[test]
     fn create_tempfile() {
-        super::new_file().unwrap();
+        let _ = super::new_file().unwrap();
     }
 
     #[test]
@@ -145,7 +147,7 @@ fn use_tempfile() {
 
     #[test]
     fn create_tempdir() {
-        super::new_dir().unwrap();
+        let _ = super::new_dir().unwrap();
     }
 
     #[test]

crates/util/Cargo.toml

@@ -1,13 +1,16 @@
 [package]
 name = "fish-util"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 
 [dependencies]
+errno.workspace = true
 fish-widestring.workspace = true
+libc.workspace = true
+nix.workspace = true
 rand.workspace = true
 
 [lints]

crates/util/src/lib.rs

@@ -1,9 +1,15 @@
 //! Generic utilities library.
 
+use errno::errno;
 use fish_widestring::prelude::*;
 use rand::{SeedableRng as _, rngs::SmallRng};
-use std::cmp::Ordering;
-use std::time;
+use std::{
+    cmp::Ordering,
+    ffi::CStr,
+    io::Write as _,
+    os::fd::{BorrowedFd, RawFd},
+    time,
+};
 
 /// Compares two wide character strings with an (arguably) intuitive ordering. This function tries
 /// to order strings in a way which is intuitive to humans with regards to sorting strings
@@ -57,14 +63,23 @@ pub fn wcsfilecmp(a: &wstr, b: &wstr) -> Ordering {
             continue;
         }
 
-        // Sort dashes after Z - see #5634
-        let mut acl = if ac == '-' { '[' } else { ac };
-        let mut bcl = if bc == '-' { '[' } else { bc };
+        let transform = |c| {
+            // Sort dashes after Z - see #5634
+            if c == '-' {
+                return '[';
+            }
+            if c == '/' {
+                return '\0';
+            }
+            c
+        };
+        let ac = transform(ac);
+        let bc = transform(bc);
         // TODO Compare the tail (enabled by Rust's Unicode support).
-        acl = acl.to_uppercase().next().unwrap();
-        bcl = bcl.to_uppercase().next().unwrap();
+        let ac = ac.to_uppercase().next().unwrap();
+        let bc = bc.to_uppercase().next().unwrap();
 
-        match acl.cmp(&bcl) {
+        match ac.cmp(&bc) {
             Ordering::Equal => {
                 ai += 1;
                 bi += 1;
@@ -127,9 +142,9 @@ pub fn wcsfilecmp_glob(a: &wstr, b: &wstr) -> Ordering {
         }
 
         // TODO Compare the tail (enabled by Rust's Unicode support).
-        let acl = ac.to_lowercase().next().unwrap();
-        let bcl = bc.to_lowercase().next().unwrap();
-        match acl.cmp(&bcl) {
+        let ac = ac.to_lowercase().next().unwrap();
+        let bc = bc.to_lowercase().next().unwrap();
+        match ac.cmp(&bc) {
             Ordering::Equal => {
                 ai += 1;
                 bi += 1;
@@ -234,6 +249,26 @@ fn wcsfilecmp_leading_digits(a: &wstr, b: &wstr) -> (Ordering, usize, usize) {
     (ret, ai, bi)
 }
 
+pub fn write_to_fd(input: &[u8], fd: RawFd) -> nix::Result<usize> {
+    nix::unistd::write(unsafe { BorrowedFd::borrow_raw(fd) }, input)
+}
+
+/// Prints the provided string, followed by a colon, space, and the string representation of the
+/// current errno via [`libc::strerror`].
+pub fn perror(s: &str) {
+    let e = errno().0;
+    let mut stderr = std::io::stderr().lock();
+    if !s.is_empty() {
+        let _ = write!(stderr, "{s}: ");
+    }
+    let slice = unsafe {
+        let msg = libc::strerror(e);
+        CStr::from_ptr(msg).to_bytes()
+    };
+    let _ = stderr.write_all(slice);
+    let _ = stderr.write_all(b"\n");
+}
+
 #[cfg(test)]
 mod tests {
     use super::wcsfilecmp;
@@ -288,5 +323,8 @@ macro_rules! validate {
         validate!("a00b", "a0b", Ordering::Less);
         validate!("a0b", "a00b", Ordering::Greater);
         validate!("a-b", "azb", Ordering::Greater);
+        validate!("a", "a b", Ordering::Less);
+        validate!("a/", "a b/", Ordering::Less);
+        validate!("a/b", "a b", Ordering::Less); // Note this is arbitrary.
     }
 }

crates/wcstringutil/Cargo.toml

@@ -1,13 +1,12 @@
 [package]
 name = "fish-wcstringutil"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 
 [dependencies]
-fish-common.workspace = true
 fish-fallback.workspace = true
 fish-widestring.workspace = true
 

crates/wcstringutil/src/lib.rs

@@ -1,8 +1,7 @@
 //! Helper functions for working with wcstring.
 
-use fish_common::{get_ellipsis_char, get_ellipsis_str};
 use fish_fallback::{fish_wcwidth, lowercase, lowercase_rev, wcscasecmp, wcscasecmp_fuzzy};
-use fish_widestring::{decode_byte_from_char, prelude::*};
+use fish_widestring::{ELLIPSIS_CHAR, prelude::*};
 
 /// Return the number of newlines in a string.
 pub fn count_newlines(s: &wstr) -> usize {
@@ -336,30 +335,6 @@ pub fn string_fuzzy_match_string(
     StringFuzzyMatch::try_create(string, match_against, anchor_start)
 }
 
-/// Implementation of wcs2bytes that accepts a callback.
-/// The first argument can be either a `&str` or `&wstr`.
-/// This invokes `func` with byte slices containing the UTF-8 encoding of the characters in the
-/// input, doing one invocation per character.
-/// If `func` returns false, it stops; otherwise it continues.
-/// Return false if the callback returned false, otherwise true.
-pub fn str2bytes_callback(input: impl IntoCharIter, mut func: impl FnMut(&[u8]) -> bool) -> bool {
-    // A `char` represents an Unicode scalar value, which takes up at most 4 bytes when encoded in UTF-8.
-    let mut converted = [0_u8; 4];
-
-    for c in input.chars() {
-        let bytes = if let Some(byte) = decode_byte_from_char(c) {
-            converted[0] = byte;
-            &converted[..=0]
-        } else {
-            c.encode_utf8(&mut converted).as_bytes()
-        };
-        if !func(bytes) {
-            return false;
-        }
-    }
-    true
-}
-
 /// Split a string by runs of any of the separator characters provided in `seps`.
 /// Note the delimiters are the characters in `seps`, not `seps` itself.
 /// `seps` may contain the NUL character.
@@ -473,48 +448,43 @@ pub fn split_about<'haystack>(
     output
 }
 
-#[derive(Eq, PartialEq)]
-pub enum EllipsisType {
-    None,
-    // Prefer niceness over minimalness
-    Prettiest,
-    // Make every character count ($ instead of ...)
-    Shortest,
-}
-
-pub fn truncate(input: &wstr, max_len: usize, etype: Option<EllipsisType>) -> WString {
-    let etype = etype.unwrap_or(EllipsisType::Prettiest);
+// TODO: This should work on render width rather than the number of codepoints.
+pub fn truncate(input: &wstr, max_len: usize) -> WString {
     if input.len() <= max_len {
         return input.to_owned();
     }
-
-    if etype == EllipsisType::None {
-        return input[..max_len].to_owned();
-    }
-    if etype == EllipsisType::Prettiest {
-        let ellipsis_str = get_ellipsis_str();
-        let mut output = input[..max_len - ellipsis_str.len()].to_owned();
-        output += ellipsis_str;
-        return output;
-    }
     let mut output = input[..max_len - 1].to_owned();
-    output.push(get_ellipsis_char());
+    output.push(ELLIPSIS_CHAR);
     output
 }
 
-pub fn trim(input: WString, any_of: Option<&wstr>) -> WString {
+fn trim_indices(input: &wstr, any_of: Option<&wstr>) -> std::ops::Range<usize> {
     let any_of = any_of.unwrap_or(L!("\t\x0B \r\n"));
-    let mut result = input;
+    let result = input;
     let Some(suffix) = result.chars().rposition(|c| !any_of.contains(c)) else {
-        return WString::new();
+        return 0..0;
     };
-    result.truncate(suffix + 1);
-
     let prefix = result
         .chars()
         .position(|c| !any_of.contains(c))
         .expect("Should have one non-trimmed character");
-    result.split_off(prefix)
+    prefix..(suffix + 1)
+}
+
+// Remove leading and trailing characters in `any_of` from the string.
+// By default, trim whitespace.
+pub fn trim<'a>(input: &'a wstr, any_of: Option<&wstr>) -> &'a wstr {
+    let range = trim_indices(input, any_of);
+    &input[range]
+}
+
+// Remove leading and trailing characters in `any_of` from the string.
+// By default, trim whitespace.
+// This trims in-place.
+pub fn trim_in_place(input: &mut WString, any_of: Option<&wstr>) {
+    let range = trim_indices(input, any_of);
+    input.truncate(range.end);
+    input.drain(0..range.start);
 }
 
 /// Return the number of escaping backslashes before a character.
@@ -565,12 +535,12 @@ fn next(&mut self) -> Option<Self::Item> {
     }
 }
 
-/// Like fish_wcwidth, but returns 0 for characters with no real width instead of -1.
+/// Like fish_wcwidth, but returns 0 for characters with no real width instead of none.
 pub fn fish_wcwidth_visible(c: char) -> isize {
     if c == '\x08' {
         return -1;
     }
-    fish_wcwidth(c).max(0)
+    fish_wcwidth(c).unwrap_or_default().try_into().unwrap()
 }
 
 #[cfg(test)]
@@ -578,7 +548,7 @@ mod tests {
     use super::{
         CaseSensitivity, ContainType, LineIterator, count_newlines, ifind, join_strings,
         split_string_tok, string_fuzzy_match_string, string_prefixes_string_case_insensitive,
-        string_suffixes_string_case_insensitive,
+        string_suffixes_string_case_insensitive, trim, trim_in_place,
     };
     use fish_widestring::prelude::*;
 
@@ -791,4 +761,20 @@ fn test_count_newlines() {
         assert_eq!(count_newlines(L!("\n")), 1);
         assert_eq!(count_newlines(L!("\n\n")), 2);
     }
+
+    #[test]
+    fn test_trim() {
+        fn test_trim(input: &wstr, any_of: Option<&wstr>, expect: &wstr) {
+            assert_eq!(trim(input, any_of), expect);
+
+            let mut s = input.to_owned();
+            trim_in_place(&mut s, any_of);
+            assert_eq!(s, expect);
+        }
+        test_trim(L!("foo"), None, L!("foo"));
+        test_trim(L!("fooff"), Some(L!("f")), L!("oo"));
+        test_trim(L!("  foo  "), None, L!("foo"));
+        test_trim(L!(""), None, L!(""));
+        test_trim(L!("  \n\n\n"), None, L!(""));
+    }
 }

crates/wgetopt/Cargo.toml

@@ -1,8 +1,8 @@
 [package]
 name = "fish-wgetopt"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 

crates/widecharwidth/Cargo.toml

@@ -1,8 +1,8 @@
 [package]
 name = "fish-widecharwidth"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license = "CC0-1.0"
 

crates/widestring/Cargo.toml

@@ -1,12 +1,14 @@
 [package]
 name = "fish-widestring"
+version = "0.0.0"
 edition.workspace = true
 rust-version.workspace = true
-version = "0.0.0"
 repository.workspace = true
 license.workspace = true
 
 [dependencies]
+libc.workspace = true
+unicode-width.workspace = true
 widestring.workspace = true
 
 [lints]

crates/widestring/src/lib.rs

@@ -6,13 +6,34 @@
 
 pub mod word_char;
 
-use std::{iter, slice};
+use std::{
+    ffi::{CStr, CString, OsStr, OsString},
+    iter,
+    os::unix::ffi::{OsStrExt as _, OsStringExt as _},
+    slice,
+};
 pub use widestring::{Utf32Str as wstr, Utf32String as WString, utf32str as L, utfstr::CharsUtf32};
 
 pub mod prelude {
     pub use crate::{IntoCharIter, L, ToWString, WExt, WString, wstr};
 }
 
+// Highest legal ASCII value.
+pub const ASCII_MAX: char = 127 as char;
+
+// Highest legal 16-bit Unicode value.
+pub const UCS2_MAX: char = '\u{FFFF}';
+
+// Highest legal byte value.
+pub const BYTE_MAX: char = 0xFF as char;
+
+// Unicode BOM value.
+pub const UTF8_BOM_WCHAR: char = '\u{FEFF}';
+
+/// The character to use where the text has been truncated.
+pub const ELLIPSIS_CHAR: char = '\u{2026}'; // ('…')
+
+pub const SPECIAL_KEY_ENCODE_BASE: char = '\u{F500}';
 // These are in the Unicode private-use range. We really shouldn't use this
 // range but have little choice in the matter given how our lexer/parser works.
 // We can't use non-characters for these two ranges because there are only 66 of
@@ -25,9 +46,79 @@ pub mod prelude {
 // Note: We don't use the highest 8 bit range (0xF800 - 0xF8FF) because we know
 // of at least one use of a codepoint in that range: the Apple symbol (0xF8FF)
 // on Mac OS X. See http://www.unicode.org/faq/private_use.html.
-pub const ENCODE_DIRECT_BASE: char = '\u{F600}';
+pub const ENCODE_DIRECT_BASE: char = char_offset(SPECIAL_KEY_ENCODE_BASE, 256);
 pub const ENCODE_DIRECT_END: char = char_offset(ENCODE_DIRECT_BASE, 256);
 
+// Use Unicode "non-characters" for internal characters as much as we can. This
+// gives us 32 "characters" for internal use that we can guarantee should not
+// appear in our input stream. See http://www.unicode.org/faq/private_use.html.
+pub const RESERVED_CHAR_BASE: char = '\u{FDD0}';
+pub const RESERVED_CHAR_END: char = '\u{FDF0}';
+// Split the available non-character values into two ranges to ensure there are
+// no conflicts among the places we use these special characters.
+pub const EXPAND_RESERVED_BASE: char = RESERVED_CHAR_BASE;
+pub const EXPAND_RESERVED_END: char = char_offset(EXPAND_RESERVED_BASE, 16);
+pub const WILDCARD_RESERVED_BASE: char = EXPAND_RESERVED_END;
+pub const WILDCARD_RESERVED_END: char = char_offset(WILDCARD_RESERVED_BASE, 16);
+// Make sure the ranges defined above don't exceed the range for non-characters.
+// This is to make sure we didn't do something stupid in subdividing the
+// Unicode range for our needs.
+const _: () = assert!(WILDCARD_RESERVED_END <= RESERVED_CHAR_END);
+
+/// Character representing any character except '/' (slash).
+pub const ANY_CHAR: char = char_offset(WILDCARD_RESERVED_BASE, 0);
+/// Character representing any character string not containing '/' (slash).
+pub const ANY_STRING: char = char_offset(WILDCARD_RESERVED_BASE, 1);
+/// Character representing any character string.
+pub const ANY_STRING_RECURSIVE: char = char_offset(WILDCARD_RESERVED_BASE, 2);
+/// This is a special pseudo-char that is not used other than to mark the
+/// end of the special characters so we can sanity check the enum range.
+#[allow(dead_code)]
+pub const ANY_SENTINEL: char = char_offset(WILDCARD_RESERVED_BASE, 3);
+
+/// Character representing a home directory.
+pub const HOME_DIRECTORY: char = char_offset(EXPAND_RESERVED_BASE, 0);
+/// Character representing process expansion for %self.
+pub const PROCESS_EXPAND_SELF: char = char_offset(EXPAND_RESERVED_BASE, 1);
+/// Character representing variable expansion.
+pub const VARIABLE_EXPAND: char = char_offset(EXPAND_RESERVED_BASE, 2);
+/// Character representing variable expansion into a single element.
+pub const VARIABLE_EXPAND_SINGLE: char = char_offset(EXPAND_RESERVED_BASE, 3);
+/// Character representing the start of a bracket expansion.
+pub const BRACE_BEGIN: char = char_offset(EXPAND_RESERVED_BASE, 4);
+/// Character representing the end of a bracket expansion.
+pub const BRACE_END: char = char_offset(EXPAND_RESERVED_BASE, 5);
+/// Character representing separation between two bracket elements.
+pub const BRACE_SEP: char = char_offset(EXPAND_RESERVED_BASE, 6);
+/// Character that takes the place of any whitespace within non-quoted text in braces
+pub const BRACE_SPACE: char = char_offset(EXPAND_RESERVED_BASE, 7);
+/// Separate subtokens in a token with this character.
+pub const INTERNAL_SEPARATOR: char = char_offset(EXPAND_RESERVED_BASE, 8);
+/// Character representing an empty variable expansion. Only used transitively while expanding
+/// variables.
+pub const VARIABLE_EXPAND_EMPTY: char = char_offset(EXPAND_RESERVED_BASE, 9);
+
+const _: () = assert!(
+    EXPAND_RESERVED_END as u32 > VARIABLE_EXPAND_EMPTY as u32,
+    "Characters used in expansions must stay within private use area"
+);
+
+/// The string represented by PROCESS_EXPAND_SELF
+pub const PROCESS_EXPAND_SELF_STR: &wstr = L!("%self");
+
+/// Return true if the character is in a range reserved for fish's private use.
+///
+/// NOTE: This is used when tokenizing the input. It is also used when reading input, before
+/// tokenization, to replace such chars with REPLACEMENT_WCHAR if they're not part of a quoted
+/// string. We don't want external input to be able to feed reserved characters into our
+/// lexer/parser or code evaluator.
+//
+// TODO: Actually implement the replacement as documented above.
+pub fn fish_reserved_codepoint(c: char) -> bool {
+    (c >= RESERVED_CHAR_BASE && c < RESERVED_CHAR_END)
+        || (c >= SPECIAL_KEY_ENCODE_BASE && c < ENCODE_DIRECT_END)
+}
+
 /// Encode a literal byte in a UTF-32 character. This is required for e.g. the echo builtin, whose
 /// escape sequences can be used to construct raw byte sequences which are then interpreted as e.g.
 /// UTF-8 by the terminal. If we were to interpret each of those bytes as a codepoint and encode it
@@ -40,6 +131,86 @@ pub fn encode_byte_to_char(byte: u8) -> char {
         .expect("private-use codepoint should be valid char")
 }
 
+/// Returns a newly allocated multibyte character string equivalent of the specified wide character
+/// string.
+///
+/// This function decodes illegal character sequences in a reversible way using the private use
+/// area.
+pub fn wcs2bytes(input: impl IntoCharIter) -> Vec<u8> {
+    let mut result = vec![];
+    wcs2bytes_appending(&mut result, input);
+    result
+}
+
+pub fn wcs2osstring(input: &wstr) -> OsString {
+    if input.is_empty() {
+        return OsString::new();
+    }
+
+    let mut result = vec![];
+    wcs2bytes_appending(&mut result, input);
+    OsString::from_vec(result)
+}
+
+/// Same as [`wcs2bytes`]. Meant to be used when we need a zero-terminated string to feed legacy APIs.
+/// Note: if `input` contains any interior NUL bytes, the result will be truncated at the first!
+pub fn wcs2zstring(input: &wstr) -> CString {
+    if input.is_empty() {
+        return CString::default();
+    }
+
+    let mut vec = Vec::with_capacity(input.len() + 1);
+    str2bytes_callback(input, |buff| {
+        vec.extend_from_slice(buff);
+        true
+    });
+    vec.push(b'\0');
+
+    match CString::from_vec_with_nul(vec) {
+        Ok(cstr) => cstr,
+        Err(err) => {
+            // `input` contained a NUL in the middle; we can retrieve `vec`, though
+            let mut vec = err.into_bytes();
+            let pos = vec.iter().position(|c| *c == b'\0').unwrap();
+            vec.truncate(pos + 1);
+            // Safety: We truncated after the first NUL
+            unsafe { CString::from_vec_with_nul_unchecked(vec) }
+        }
+    }
+}
+
+/// Like [`wcs2bytes`], but appends to `output` instead of returning a new string.
+pub fn wcs2bytes_appending(output: &mut Vec<u8>, input: impl IntoCharIter) {
+    str2bytes_callback(input, |buff| {
+        output.extend_from_slice(buff);
+        true
+    });
+}
+
+/// Implementation of wcs2bytes that accepts a callback.
+/// The first argument can be either a `&str` or `&wstr`.
+/// This invokes `func` with byte slices containing the UTF-8 encoding of the characters in the
+/// input, doing one invocation per character.
+/// If `func` returns false, it stops; otherwise it continues.
+/// Return false if the callback returned false, otherwise true.
+pub fn str2bytes_callback(input: impl IntoCharIter, mut func: impl FnMut(&[u8]) -> bool) -> bool {
+    // A `char` represents an Unicode scalar value, which takes up at most 4 bytes when encoded in UTF-8.
+    let mut converted = [0_u8; 4];
+
+    for c in input.chars() {
+        let bytes = if let Some(byte) = decode_byte_from_char(c) {
+            converted[0] = byte;
+            &converted[..=0]
+        } else {
+            c.encode_utf8(&mut converted).as_bytes()
+        };
+        if !func(bytes) {
+            return false;
+        }
+    }
+    true
+}
+
 /// Decode a literal byte from a UTF-32 character.
 pub fn decode_byte_from_char(c: char) -> Option<u8> {
     if c >= ENCODE_DIRECT_BASE && c < ENCODE_DIRECT_END {
@@ -53,6 +224,278 @@ pub fn decode_byte_from_char(c: char) -> Option<u8> {
     }
 }
 
+/// A trait to make it more convenient to pass ascii/Unicode strings to functions that can take
+/// non-Unicode values. The result is nul-terminated and can be passed to OS functions.
+///
+/// This is only implemented for owned types where an owned instance will skip allocations (e.g.
+/// `CString` can return `self`) but not implemented for owned instances where a new allocation is
+/// always required (e.g. implemented for `&wstr` but not `WideString`) because you might as well be
+/// left with the original item if we're going to allocate from scratch in all cases.
+pub trait ToCString {
+    /// Correctly convert to a nul-terminated [`CString`] that can be passed to OS functions.
+    fn to_cstring(self) -> CString;
+}
+
+impl ToCString for CString {
+    fn to_cstring(self) -> CString {
+        self
+    }
+}
+
+impl ToCString for &CStr {
+    fn to_cstring(self) -> CString {
+        self.to_owned()
+    }
+}
+
+/// Safely converts from `&wstr` to a `CString` to a nul-terminated `CString` that can be passed to
+/// OS functions, taking into account non-Unicode values that have been shifted into the private-use
+/// range by using [`wcs2zstring()`].
+impl ToCString for &wstr {
+    /// The wide string may contain non-Unicode bytes mapped to the private-use Unicode range, so we
+    /// have to use [`wcs2zstring()`](self::wcs2zstring) to convert it correctly.
+    fn to_cstring(self) -> CString {
+        self::wcs2zstring(self)
+    }
+}
+
+/// Safely converts from `&WString` to a nul-terminated `CString` that can be passed to OS
+/// functions, taking into account non-Unicode values that have been shifted into the private-use
+/// range by using [`wcs2zstring()`].
+impl ToCString for &WString {
+    fn to_cstring(self) -> CString {
+        self.as_utfstr().to_cstring()
+    }
+}
+
+/// Convert a (probably ascii) string to CString that can be passed to OS functions.
+impl ToCString for Vec<u8> {
+    fn to_cstring(mut self) -> CString {
+        self.push(b'\0');
+        CString::from_vec_with_nul(self).unwrap()
+    }
+}
+
+/// Convert a (probably ascii) string to nul-terminated CString that can be passed to OS functions.
+impl ToCString for &[u8] {
+    fn to_cstring(self) -> CString {
+        CString::new(self).unwrap()
+    }
+}
+
+mod decoder {
+    use crate::{ENCODE_DIRECT_BASE, ENCODE_DIRECT_END, char_offset, wstr};
+    use buffer::Buffer;
+    use std::{char::REPLACEMENT_CHARACTER, ops::Range};
+    use widestring::utfstr::CharsUtf32;
+
+    mod buffer {
+        // The size required for a PUA-encoded character from our special PUA range - 1,
+        // since that is the maximum number characters our look-ahead needs to check.
+        const MAX_SIZE: usize = 2;
+        pub(super) struct Buffer {
+            buffer: [char; MAX_SIZE],
+            length: usize,
+        }
+
+        impl Buffer {
+            pub(super) fn empty() -> Self {
+                Self {
+                    buffer: ['\0'; MAX_SIZE],
+                    length: 0,
+                }
+            }
+
+            pub(super) fn push(&mut self, c: char) {
+                self.buffer[self.length] = c;
+                self.length += 1;
+            }
+
+            pub(super) fn pop(&mut self) -> Option<char> {
+                if self.length == 0 {
+                    return None;
+                }
+                self.length -= 1;
+                Some(self.buffer[self.length])
+            }
+
+            pub(super) fn pop_front(&mut self) -> Option<char> {
+                if self.length == 0 {
+                    return None;
+                }
+                self.buffer.rotate_left(1);
+                self.length -= 1;
+                Some(self.buffer[MAX_SIZE - 1])
+            }
+        }
+    }
+
+    const PUA_ENCODE_RANGE: Range<char> = ENCODE_DIRECT_BASE..ENCODE_DIRECT_END;
+    const ENCODED_PUA_CHAR_FIRST_CHAR: char = char_offset(ENCODE_DIRECT_BASE, 0xef);
+    // The second UTF-8 byte of a character in our special PUA range is in the range
+    // 0x98..0x9c.
+    const ENCODED_PUA_CHAR_SECOND_CHAR_RANGE: Range<char> =
+        char_offset(ENCODE_DIRECT_BASE, 0x98)..char_offset(ENCODE_DIRECT_BASE, 0x9c);
+
+    /// This serves as the data container for building a double-ended iterator which decodes our
+    /// PUA-encoded chars into a char iterator where each encoded non-UTF-8 byte is replaced by the
+    /// replacement character, and each encoded PUA codepoint is turned back into a single char
+    /// whose value is the original PUA codepoint.
+    ///
+    /// The latter part makes the decoding logic somewhat complicated, because encoded PUA chars
+    /// take up 3 chars in our encoding. Therefore, in some cases, we need to take more than 1 char
+    /// from the `encoded_chars` iterator before we know whether to decode 3 chars together into a
+    /// single char, or whether the chars should be replaced by the replacement char individually.
+    /// In cases where we took more than 1 char and then notice that individual replacement is
+    /// warranted, we return a replacement char for the first char we took from the iterator, and
+    /// cache the 1 or 2 other chars we read in `buffer_front` or `buffer_back`, depending on the
+    /// reading direction. Buffers store elements in such an order that getting the next character
+    /// requires `pop` when using the buffer associated with the current reading direction, and
+    /// `pop_front` when using the other buffer. This is done to optimize the common case of
+    /// iterating in a single direction.
+    ///
+    /// The buffers have to be considered before taking more chars from `encoded_chars`.
+    /// If the iterator is only read in one direction, the buffer for the other direction will not
+    /// be used. But because it's possible that the iterator is read from both ends, it can happen
+    /// that when `encoded_chars` runs out, the buffer for the opposite reading direction is
+    /// non-empty. In the [`Self::next`] and [`Self::next_back`] implementations, this logic is
+    /// encapsulated into closures for getting the next char from the appropriate source.
+    /// At most 2 chars will ever be stored in a buffer, so they are implemented using a fixed-size
+    /// array, requiring no heap allocations.
+    ///
+    /// Note that in most cases, we can avoid using the buffers, and simply forward the char
+    /// obtained from `encoded_chars`. Only chars in [`PUA_ENCODE_RANGE`] can possibly encode PUA
+    /// chars, so if we read any other char, we know that it's not part of such an encoding and can
+    /// return it directly. If we read in the forward direction, we can also exploit knowledge about
+    /// the possible values of our PUA encoding. Specifically, the first char in such an encoding
+    /// will always be [`ENCODED_PUA_CHAR_FIRST_CHAR`], and the second char will be in the range
+    /// [`ENCODED_PUA_CHAR_SECOND_CHAR_RANGE`].
+    pub(super) struct Decoder<'a> {
+        encoded_chars: CharsUtf32<'a>,
+        buffer_front: Buffer,
+        buffer_back: Buffer,
+    }
+
+    impl<'a> Decoder<'a> {
+        pub(super) fn new(encoded_str: &'a wstr) -> Self {
+            Self {
+                encoded_chars: encoded_str.chars(),
+                buffer_front: Buffer::empty(),
+                buffer_back: Buffer::empty(),
+            }
+        }
+    }
+
+    fn try_pua_decoding(encoding: &[char; 3]) -> Option<char> {
+        let mut bytes = [0u8; 3];
+        for (index, &c) in encoding.iter().enumerate() {
+            bytes[index] = super::decode_byte_from_char(c)?;
+        }
+        let first_decoded_char =
+            std::str::from_utf8(&bytes).ok()?.chars().next().expect(
+                "Non-empty byte slice which is valid UTF-8 must result in at least one char.",
+            );
+        // For strings whose width we compute, we only expect invalid UTF-8 and codepoints from the
+        // PUA encoding range to be PUA encoded.
+        // If we reach this point, the encoded bytes are valid UTF-8, so the only remaining
+        // expected case are codepoints from the PUA encoding range.
+        // These all take 3 bytes to represent in UTF-8, so if we check that the first parsed
+        // codepoint is in the expected range, we know that exactly 3 bytes were consumed for
+        // parsing this codepoint.
+        assert!(PUA_ENCODE_RANGE.contains(&first_decoded_char));
+        Some(first_decoded_char)
+    }
+
+    fn replace_if_pua_encoded(c: char) -> char {
+        if PUA_ENCODE_RANGE.contains(&c) {
+            REPLACEMENT_CHARACTER
+        } else {
+            c
+        }
+    }
+
+    impl<'a> Iterator for Decoder<'a> {
+        type Item = char;
+
+        fn next(&mut self) -> Option<Self::Item> {
+            let mut get_next_char = || {
+                self.buffer_front
+                    .pop()
+                    .or_else(|| self.encoded_chars.next())
+                    .or_else(|| self.buffer_back.pop_front())
+            };
+            let c_0 = get_next_char()?;
+            if c_0 != ENCODED_PUA_CHAR_FIRST_CHAR {
+                return Some(replace_if_pua_encoded(c_0));
+            }
+            if let Some(c_1) = get_next_char() {
+                if ENCODED_PUA_CHAR_SECOND_CHAR_RANGE.contains(&c_1) {
+                    if let Some(c_2) = get_next_char() {
+                        if let Some(decoded_pua_char) = try_pua_decoding(&[c_0, c_1, c_2]) {
+                            return Some(decoded_pua_char);
+                        }
+                        self.buffer_front.push(c_2);
+                    }
+                }
+                self.buffer_front.push(c_1);
+            }
+            // If decoding 3 consecutive PUA chars into the encoded PUA char fails, `c_0` should be
+            // returned. `c_0` is `ENCODED_PUA_CHAR_FIRST_CHAR` if we reach this point, so return
+            // the `REPLACEMENT_CHARACTER` in these cases.
+            Some(REPLACEMENT_CHARACTER)
+        }
+    }
+
+    impl<'a> DoubleEndedIterator for Decoder<'a> {
+        fn next_back(&mut self) -> Option<Self::Item> {
+            let mut get_next_char = || {
+                self.buffer_back
+                    .pop()
+                    .or_else(|| self.encoded_chars.next_back())
+                    .or_else(|| self.buffer_front.pop_front())
+            };
+            let c_2 = get_next_char()?;
+            if !PUA_ENCODE_RANGE.contains(&c_2) {
+                return Some(c_2);
+            }
+            if let Some(c_1) = get_next_char() {
+                if PUA_ENCODE_RANGE.contains(&c_1) {
+                    if let Some(c_0) = get_next_char() {
+                        if let Some(decoded_pua_char) = try_pua_decoding(&[c_0, c_1, c_2]) {
+                            return Some(decoded_pua_char);
+                        }
+                        self.buffer_back.push(c_0);
+                    }
+                    self.buffer_back.push(c_1);
+                }
+            }
+            // If decoding 3 consecutive PUA chars into the encoded PUA char fails, `c_2` should be
+            // returned. `c_2` is in `PUA_ENCODE_RANGE` if we reach this point, so return the
+            // `REPLACEMENT_CHARACTER` in these cases.
+            Some(REPLACEMENT_CHARACTER)
+        }
+    }
+}
+
+/// Only exists for tests. Exported because the encoding functionality is not available in this
+/// crate. Do not use for non-testing purposes.
+pub fn decode_with_replacement(encoded_str: &wstr) -> impl DoubleEndedIterator<Item = char> {
+    decoder::Decoder::new(encoded_str)
+}
+
+/// Takes a PUA-encoded string, decodes it by restoring encoded PUA codepoints and replacing encoded
+/// non-UTF-8 bytes by the replacement character U+FFFD.
+/// The result is passed to the [`unicode_width`] crate, which will compute its width, which will be
+/// the return value of this function.
+pub fn decoded_width(encoded_str: &wstr) -> usize {
+    // TODO: Avoid constructing String by using `unicode_width::char_iter_width` once that is
+    // available in a released version of the crate (it's already on the crate's master branch).
+    use unicode_width::UnicodeWidthStr as _;
+    decoder::Decoder::new(encoded_str)
+        .collect::<String>()
+        .width()
+}
+
 pub const fn char_offset(base: char, offset: u32) -> char {
     match char::from_u32(base as u32 + offset) {
         Some(c) => c,
@@ -60,6 +503,82 @@ pub const fn char_offset(base: char, offset: u32) -> char {
     }
 }
 
+/// Encodes the bytes in `input` into a [`WString`], encoding non-UTF-8 bytes into private-use-area
+/// code-points. Bytes which would be parsed into our reserved PUA range are encoded individually,
+/// to allow for correct round-tripping.
+pub fn bytes2wcstring(mut input: &[u8]) -> WString {
+    if input.is_empty() {
+        return WString::new();
+    }
+
+    let mut result = WString::with_capacity(input.len());
+
+    fn append_escaped_str(output: &mut WString, input: &str) {
+        for (i, c) in input.char_indices() {
+            if fish_reserved_codepoint(c) {
+                for byte in &input.as_bytes()[i..i + c.len_utf8()] {
+                    output.push(encode_byte_to_char(*byte));
+                }
+            } else {
+                output.push(c);
+            }
+        }
+    }
+
+    while !input.is_empty() {
+        match std::str::from_utf8(input) {
+            Ok(parsed_str) => {
+                append_escaped_str(&mut result, parsed_str);
+                // The entire remaining input could be parsed, so we are done.
+                break;
+            }
+            Err(e) => {
+                let (valid, after_valid) = input.split_at(e.valid_up_to());
+                // SAFETY: The previous `str::from_utf8` call established that the prefix `valid`
+                // is valid UTF-8. This prefix may be empty.
+                let parsed_str = unsafe { std::str::from_utf8_unchecked(valid) };
+                append_escaped_str(&mut result, parsed_str);
+                // The length of the prefix of `after_valid` which is invalid UTF-8.
+                // The remaining bytes of `input` (if any) will be parsed in subsequent iterations
+                // of the loop, starting from the first byte that starts a valid UTF-8-encoded codepoint.
+                // `error_len` can return `None`, if it sees a byte sequence that could be the
+                // prefix of a valid code-point encoding at the end of the byte slice.
+                // This is useful when the input is chunked, but we don't do that, so in this case
+                // we use our custom encoding for all remaining bytes (at most 3).
+                let error_len = e.error_len().unwrap_or(after_valid.len());
+                for byte in &after_valid[..error_len] {
+                    result.push(encode_byte_to_char(*byte));
+                }
+                input = &after_valid[error_len..];
+            }
+        }
+    }
+    result
+}
+
+/// Use this rather than [`WString::from_str`] when the input could contain PUA bytes we use to
+/// encode non-UTF-8 bytes. Otherwise, when decoding the resulting [`WString`], the PUA bytes in
+/// the input would be converted to non-UTF-8 bytes.
+pub fn str2wcstring<S: AsRef<str>>(input: S) -> WString {
+    bytes2wcstring(input.as_ref().as_bytes())
+}
+
+pub fn cstr2wcstring<C: AsRef<CStr>>(input: C) -> WString {
+    bytes2wcstring(input.as_ref().to_bytes())
+}
+
+pub fn osstr2wcstring<O: AsRef<OsStr>>(input: O) -> WString {
+    bytes2wcstring(input.as_ref().as_bytes())
+}
+
+/// # SAFETY
+///
+/// `input` must point to a valid NUL-terminated string.
+pub unsafe fn charptr2wcstring(input: *const libc::c_char) -> WString {
+    let input: &[u8] = unsafe { CStr::from_ptr(input).to_bytes() };
+    bytes2wcstring(input)
+}
+
 /// Finds `needle` in a `haystack` and returns the index of the first matching element, if any.
 ///
 /// # Examples
@@ -89,6 +608,12 @@ pub trait ToWString {
     fn to_wstring(&self) -> WString;
 }
 
+impl ToWString for std::path::Path {
+    fn to_wstring(&self) -> WString {
+        bytes2wcstring(self.as_os_str().as_encoded_bytes())
+    }
+}
+
 #[inline]
 fn to_wstring_impl(mut val: u64, neg: bool) -> WString {
     // 20 digits max in u64: 18446744073709551616.

crates/xtask/Cargo.toml

@@ -1,11 +1,21 @@
 [package]
 name = "xtask"
 version = "0.0.0"
-rust-version.workspace = true
 edition.workspace = true
+rust-version.workspace = true
 repository.workspace = true
 
 [dependencies]
+anstyle.workspace = true
+anyhow.workspace = true
 clap.workspace = true
+clap_complete.workspace = true
 fish-build-helper.workspace = true
+fish-common.workspace = true
 fish-tempfile.workspace = true
+fish-widestring.workspace = true
+fluent-ftl-tools.workspace = true
+fluent-syntax.workspace = true
+ignore.workspace = true
+pcre2.workspace = true
+walkdir.workspace = true

crates/xtask/src/fluent.rs

@@ -0,0 +1,434 @@
+use crate::files_with_extension;
+use anyhow::{Context, Result, anyhow, bail};
+use clap::{Args, Subcommand};
+use fish_build_helper::{default_ftl_file, ftl_dir};
+use fluent_ftl_tools::{
+    HasEntries,
+    annotate::{
+        Annotation, MessageAnnotationRead, MessageAnnotationWrite, add_annotation_to_messages,
+        get_message_ids_with_different_value, modify_messages, remove_annotation_from_messages,
+    },
+    consistency::{check_all_resource_files, remove_inconsistent_translations},
+    delete_message_from_paths, filter_resource_messages,
+    format::{FormattingMode, format_text},
+    format_resource,
+    missing::find_missing_message_ids_in_files,
+    parse_as_syntax_resource, parse_str_as_syntax_resource,
+    rename::rename_in_all_files,
+    serialize_resource, serialize_resource_to_file, serialize_resources_to_files,
+};
+use fluent_syntax::{
+    ast::{Entry, Resource},
+    serializer::Serializer,
+};
+use std::{
+    collections::{HashMap, HashSet},
+    io::{Read, Write},
+    path::{Path, PathBuf},
+    process::{Command, Stdio},
+};
+
+#[derive(Subcommand)]
+pub enum FluentCommandArgs {
+    /// Check consistency of FTL files.
+    Check {
+        /// Also check if `en.ftl` is up to date with respect to the Rust sources.
+        /// This argument optionally takes a value, which must be the path to the directory into
+        /// which the message definitions have been extracted.
+        /// If this argument is provided without a value, fish will be recompiled with the `fluent-extract` feature.
+        /// See <https://github.com/fish-shell/fish-shell/pull/11928#discussion_r3379980617>
+        #[arg(long)]
+        from_source: Option<Option<PathBuf>>,
+        files: Vec<PathBuf>,
+    },
+    /// Format files. Without arguments, all FTL files are formatted.
+    /// If '-' is specified, input is read from stdin and output written to stdout.
+    Format { files: Vec<PathBuf> },
+    /// Rename an ID or variables used with an ID.
+    Rename(RenameArgs),
+    /// Resolve outdated translations.
+    /// The `update` subcommand of this xtask adds annotations to outdated translations.
+    /// These can be resolved manually, but for developers not familiar with the affected languages
+    /// can be more convenient to use this tool.
+    #[command(subcommand)]
+    ResolveOutdated(ResolveOutdatedCommand),
+    /// Display message IDs which could be added to the specified file(s), or all files if no
+    /// arguments are given.
+    ShowMissing { files: Vec<PathBuf> },
+    /// Generate en.ftl from the source code and update translations.
+    /// New messages will silently be inserted into en.ftl.
+    /// Deleted messages will be removed from all FTL files.
+    /// If a Fluent variable is renamed or removed, the translations of the message will be deleted.
+    /// For other changes to a message, developer interaction is required.
+    /// All affected translations will be annotated to indicate that they are outdated.
+    /// Our checks will fail as long as such annotations are present.
+    /// Developers have several options to address them, depending on the situation.
+    /// If the developer is familiar with the language of the translation, they can manually update
+    /// the translation as required and removed the annotation.
+    /// Otherwise, `cargo xtask fluent resolve-outdated` can be used.
+    /// See its description for details.
+    Update {
+        /// Path to the directory into which the message definitions from the Rust sources have been extracted.
+        /// If this is not specified, fish will be compiled with the `fluent-extract` feature to
+        /// obtain the message definitions.
+        #[arg(long)]
+        extraction_dir: Option<PathBuf>,
+    },
+}
+
+#[derive(Args)]
+pub struct RenameArgs {
+    /// The message ID.
+    /// Use 'old=new', where 'old' is the old message ID as currently present in the file,
+    /// and 'new' the name it should be renamed to.
+    /// If the ID should not change, use 'old'.
+    id: String,
+    /// The variables to rename for the specified ID.
+    /// Use 'old=new', where 'old' is the old variable name as currently present in the file,
+    /// and 'new' the name it should be renamed to.
+    vars: Vec<String>,
+}
+
+#[derive(Subcommand)]
+pub enum ResolveOutdatedCommand {
+    /// Delete the message.
+    /// Use this when the meaning of the message changed significantly.
+    Delete(ResolveOutdatedArgs),
+    /// Remove the annotation.
+    /// Use this when the translation is already up-to-date.
+    Ignore(ResolveOutdatedArgs),
+    /// Keep the message as is and replace the OUTDATED annotation with a NEEDS-REVIEW annotation to
+    /// indicate that translators should have a look at it.
+    /// Use this for minor changes to the English message, which do not change its meaning much.
+    NeedsReview(ResolveOutdatedArgs),
+    /// Show unresolved messages.
+    ShowUnresolved {
+        /// Show outdated translations for this message, instead of all messages with outdated
+        /// translations.
+        /// Can be specified multiple times.
+        #[arg(long, value_name = "Message ID")]
+        id: Vec<String>,
+        /// Consider translations for this language instead of all languages.
+        /// Can be specified multiple times.
+        /// The value must either be a path to an FTL file or the name of a language as it appears
+        /// in one of our FTL file names.
+        /// The `.ftl` suffix is optional when specifying the language name.
+        #[arg(long)]
+        language: Vec<PathBuf>,
+        /// Print the outdated message definitions.
+        #[arg(long)]
+        definitions: bool,
+    },
+}
+
+#[derive(Args)]
+pub struct ResolveOutdatedArgs {
+    /// A Fluent message ID of a message annotated as outdated.
+    message_id: String,
+    /// Paths to any number of FTL files. If none are specified, all FTL files are considered.
+    files: Vec<PathBuf>,
+}
+
+impl ResolveOutdatedArgs {
+    fn get_file_paths(&self) -> Result<Vec<PathBuf>> {
+        if self.files.is_empty() {
+            non_default_ftl_files()
+        } else {
+            Ok(self.files.clone())
+        }
+    }
+}
+
+fn non_default_ftl_files() -> Result<Vec<PathBuf>> {
+    let all_ftl_files = files_with_extension([ftl_dir()], "ftl")?;
+    let mut non_default_files = Vec::with_capacity(all_ftl_files.len() - 1);
+    let default_path = default_ftl_file().canonicalize()?;
+    for path in all_ftl_files {
+        if path.canonicalize()? != default_path {
+            non_default_files.push(path);
+        }
+    }
+    Ok(non_default_files)
+}
+
+pub fn fluent(args: FluentCommandArgs) -> Result<()> {
+    use FluentCommandArgs::*;
+    match args {
+        Check { from_source, files } => check(from_source, files),
+        Format { files } => format(files),
+        Rename(rename_args) => rename(rename_args),
+        ResolveOutdated(command) => resolve_outdated(command),
+        ShowMissing { files } => show_missing(files),
+        Update { extraction_dir } => update(extraction_dir),
+    }
+}
+
+fn check(from_source: Option<Option<PathBuf>>, mut files: Vec<PathBuf>) -> Result<()> {
+    if let Some(source) = from_source {
+        let generated_resource = generate_default_resource(source)?;
+        let mut diff_process = Command::new("diff")
+            .arg("-u")
+            .arg(default_ftl_file().as_ref())
+            .arg("-")
+            .stdin(Stdio::piped())
+            .spawn()
+            .context("Failed to run diff")?;
+        diff_process
+            .stdin
+            .take()
+            .unwrap()
+            .write_all(serialize_resource(&generated_resource).as_bytes())
+            .context("Failed to write to stdin of diff process")?;
+        let diff_status = diff_process
+            .wait()
+            .context("Failed to wait for diff child.")?;
+        if !diff_status.success() {
+            bail!(
+                "{:?} is not up to date.\n\
+                Run `cargo xtask fluent update` to resolve this.",
+                default_ftl_file()
+            );
+        }
+    }
+    if files.is_empty() {
+        files = non_default_ftl_files()?;
+    }
+    check_all_resource_files(default_ftl_file(), &files)
+}
+
+fn format(mut files: Vec<PathBuf>) -> Result<()> {
+    if files == [PathBuf::from("-")] {
+        let mut input = String::new();
+        std::io::stdin().read_to_string(&mut input).unwrap();
+        let formatted_text = format_text(&input).with_context(|| {
+            print!("{input}");
+            "Formatting input failed"
+        })?;
+        print!("{formatted_text}");
+        return Ok(());
+    }
+
+    if files.is_empty() {
+        files = files_with_extension([ftl_dir()], "ftl")?;
+    }
+    let errors = files
+        .iter()
+        .filter_map(|path| {
+            fluent_ftl_tools::format::format_path(path, FormattingMode::Rewrite).err()
+        })
+        .collect::<Vec<anyhow::Error>>();
+    if !errors.is_empty() {
+        let mut error_message = String::from("Found these errors:\n");
+        for e in errors {
+            error_message.push_str(&format!("{e}\n"));
+        }
+        bail!("{error_message}");
+    }
+    Ok(())
+}
+
+fn rename(args: RenameArgs) -> Result<()> {
+    let old_id;
+    let new_id;
+    match args.id.split_once('=') {
+        Some((old, new)) => {
+            old_id = old.into();
+            new_id = Some(new.into());
+        }
+        None => {
+            old_id = args.id;
+            new_id = None;
+        }
+    }
+    let mut variable_update = HashMap::new();
+    for arg in args.vars {
+        match arg.split_once('=') {
+            Some((old, new)) => {
+                variable_update.insert(old.into(), new.into());
+            }
+            None => {
+                bail!("Argument '{arg}' must use the format 'old=new'")
+            }
+        }
+    }
+    let files = files_with_extension([ftl_dir()], "ftl")?;
+    let resources = rename_in_all_files(&files, &old_id, &new_id, &variable_update)
+        .map_err(|e| anyhow!("Failed to perform renaming:\n{e}"))?;
+    serialize_resources_to_files(&resources).map_err(|e| anyhow!("Failed to update files:\n{e}"))
+}
+
+fn show_missing(mut files: Vec<PathBuf>) -> Result<()> {
+    if files.is_empty() {
+        files = files_with_extension([ftl_dir()], "ftl")?;
+    }
+    match find_missing_message_ids_in_files(default_ftl_file(), &files) {
+        Ok(None) => {
+            println!("No missing messages.");
+        }
+        Ok(Some(missing_message)) => {
+            println!("{missing_message}");
+        }
+        Err(e) => {
+            bail!("Error:\n{e}");
+        }
+    }
+    Ok(())
+}
+
+fn resolve_outdated(command: ResolveOutdatedCommand) -> Result<()> {
+    match command {
+        ResolveOutdatedCommand::Delete(resolve_outdated_args) => {
+            let paths = resolve_outdated_args.get_file_paths()?;
+            delete_message_from_paths(&resolve_outdated_args.message_id, &paths)
+        }
+        ResolveOutdatedCommand::Ignore(resolve_outdated_args) => {
+            let files = resolve_outdated_args.get_file_paths()?;
+            remove_annotation_from_messages(
+                Annotation::Outdated,
+                &HashSet::from_iter([resolve_outdated_args.message_id]),
+                &files,
+            )
+        }
+        ResolveOutdatedCommand::NeedsReview(resolve_outdated_args) => {
+            let files = resolve_outdated_args.get_file_paths()?;
+            modify_messages(
+                |message| {
+                    message.add_annotation(Annotation::NeedsReview)?;
+                    message.remove_annotation(Annotation::Outdated)
+                },
+                &HashSet::from_iter([resolve_outdated_args.message_id]),
+                &files,
+            )
+        }
+        ResolveOutdatedCommand::ShowUnresolved {
+            id: message_ids,
+            language: languages,
+            definitions,
+        } => {
+            let languages = if languages.is_empty() {
+                non_default_ftl_files()?
+            } else {
+                let mut language_paths = Vec::with_capacity(languages.len());
+                let ftl_dir = ftl_dir().canonicalize()?;
+                for lang in languages {
+                    if std::fs::exists(&lang)? {
+                        language_paths.push(lang);
+                        continue;
+                    }
+                    if let Some(lang_name) = lang.to_str() {
+                        let lang_name = lang_name.strip_suffix(".ftl").unwrap_or(lang_name);
+                        let file_path = ftl_dir.join(lang_name).with_extension("ftl");
+                        if std::fs::exists(&file_path)? {
+                            language_paths.push(file_path);
+                            continue;
+                        }
+                    }
+                    bail!(
+                        "Language {lang:?} is invalid. It must be a path to an FTL file or the name of a language as present in a file name of one of our FTL files."
+                    )
+                }
+                language_paths
+            };
+            let id_set: HashSet<&String> = HashSet::from_iter(&message_ids);
+            for lang in &languages {
+                println!("{lang:?}:");
+                let mut resource =
+                    filter_resource_messages(parse_as_syntax_resource(lang)?, |message| {
+                        message.has_annotation(Annotation::Outdated)
+                    })?;
+                if !id_set.is_empty() {
+                    resource = filter_resource_messages(resource, |message| {
+                        Ok(id_set.contains(&message.id.name))
+                    })?;
+                }
+                for entry in &resource.body {
+                    let Entry::Message(message) = entry else {
+                        continue;
+                    };
+                    if definitions {
+                        let mut serializer =
+                            Serializer::new(fluent_syntax::serializer::Options { with_junk: true });
+                        serializer.serialize_message(message);
+                        let serialized_message = serializer.into_serialized_text();
+                        println!("{serialized_message}");
+                    } else {
+                        println!("{}", message.id.name);
+                    }
+                }
+                println!();
+            }
+            Ok(())
+        }
+    }
+}
+
+fn update(extraction_dir: Option<PathBuf>) -> Result<()> {
+    let old_default_resource = parse_as_syntax_resource(default_ftl_file())
+        .context("Failed to parse existing default FTL file")?;
+    let generated_resource = generate_default_resource(extraction_dir)
+        .context("Failed to generate new default FTL file")?;
+    let ids_of_outdated_messages =
+        get_message_ids_with_different_value(&old_default_resource, &generated_resource);
+    let non_default_ftl_paths = non_default_ftl_files()?;
+    remove_inconsistent_translations(
+        generated_resource.all_message_vars()?,
+        &non_default_ftl_paths,
+    )?;
+    add_annotation_to_messages(
+        Annotation::Outdated,
+        &ids_of_outdated_messages,
+        &non_default_ftl_paths,
+    )?;
+    serialize_resource_to_file(&generated_resource, default_ftl_file())
+        .context("Failed to update FTL file for default language.")?;
+    Ok(())
+}
+
+fn generate_default_resource(extraction_dir: Option<PathBuf>) -> Result<Resource<String>> {
+    fn concat_unique_file_content<P: AsRef<Path>>(dir: P) -> Result<String> {
+        let mut unique_file_contents = HashSet::new();
+        for dir_entry in std::fs::read_dir(&dir)
+            .with_context(|| format!("Failed to read from directory {:?}", dir.as_ref()))?
+        {
+            let dir_entry = dir_entry
+                .with_context(|| format!("Error traversing directory {:?}", dir.as_ref()))?;
+            if dir_entry
+                .file_type()
+                .with_context(|| format!("Could not get file type for {:?}", dir_entry.path()))?
+                .is_file()
+            {
+                unique_file_contents.insert(
+                    std::fs::read_to_string(dir_entry.path()).with_context(|| {
+                        format!("Could not read from file {:?}", dir_entry.path())
+                    })?,
+                );
+            }
+        }
+        let mut concatenated_content = String::new();
+        unique_file_contents
+            .iter()
+            .for_each(|file_content| concatenated_content.push_str(file_content));
+        Ok(concatenated_content)
+    }
+
+    let id_file_content = match extraction_dir {
+        Some(dir) => concat_unique_file_content(dir).unwrap(),
+        None => {
+            let temp_dir = fish_tempfile::new_dir().unwrap();
+            Command::new(env!("CARGO"))
+                .args([
+                    "check",
+                    "--workspace",
+                    "--all-targets",
+                    "--features=fluent-extract",
+                ])
+                .env("FISH_FLUENT_EXTRACTION_DIR", temp_dir.path().as_os_str())
+                .status()
+                .map_err(|e| format!("Failed to extract Fluent IDs: {e}"))
+                .unwrap();
+            concat_unique_file_content(temp_dir.path()).unwrap()
+        }
+    };
+    let resource = parse_str_as_syntax_resource(&id_file_content).map_err(|e| anyhow!("{e}"))?;
+    format_resource(resource).map_err(|e| anyhow!("{e}"))
+}

crates/xtask/src/format.rs

@@ -0,0 +1,208 @@
+use anstyle::{AnsiColor, Style};
+use anyhow::{Context, Result, bail};
+use clap::Args;
+use std::{
+    io::{ErrorKind, Write},
+    path::PathBuf,
+    process::{Command, Stdio},
+};
+
+use crate::files_with_extension;
+
+const GREEN: Style = AnsiColor::Green.on_default();
+const YELLOW: Style = AnsiColor::Yellow.on_default();
+
+#[derive(Args)]
+pub struct FormatArgs {
+    /// Consider all eligible files.
+    #[arg(long)]
+    all: bool,
+    /// Report files which are not formatted as expected, without modifying any files.
+    #[arg(long)]
+    check: bool,
+    /// Format files even if uncommitted changes are detected.
+    #[arg(long)]
+    force: bool,
+    paths: Vec<PathBuf>,
+}
+
+pub fn format(args: FormatArgs) -> Result<()> {
+    if !args.all && args.paths.is_empty() {
+        println!(
+            "{YELLOW}warning: No paths specified. Nothing to do. Use the \"--all\" flag to consider all eligible files.{YELLOW:#}"
+        );
+        return Ok(());
+    }
+    if !args.force && !args.check {
+        match Command::new("git")
+            .args(["status", "--porcelain", "--short", "--untracked-files=all"])
+            .output()
+        {
+            Ok(output) => {
+                if !output.stdout.is_empty() {
+                    std::io::stdout()
+                        .write_all(&output.stdout)
+                        .context("Could not write to stdout.")?;
+                    print!(
+                        "You have uncommitted changes (listed above). Are you sure you want to format? (y/N): "
+                    );
+                    std::io::stdout()
+                        .flush()
+                        .context("Could not flush stdout.")?;
+                    let mut response = String::new();
+                    std::io::stdin()
+                        .read_line(&mut response)
+                        .context("Could not read from stdin.")?;
+                    if response.trim_end() != "y" {
+                        println!("Exiting without formatting.");
+                        return Ok(());
+                    }
+                }
+            }
+            Err(e) => {
+                if e.kind() == ErrorKind::NotFound {
+                    println!(
+                        "{YELLOW}warning: Did not find git, will proceed without checking for unstaged changes.{YELLOW:#}"
+                    )
+                } else {
+                    bail!("Failed to run git status:\n{e}");
+                }
+            }
+        }
+    }
+    format_fish(&args)?;
+    format_fluent(&args)?;
+    format_python(&args)?;
+    format_rust(&args)?;
+    Ok(())
+}
+
+fn run_formatter(formatter: &mut Command, name: &str) -> Result<()> {
+    println!("=== Running {GREEN}{name}{GREEN:#}");
+    match formatter.status() {
+        Ok(exit_status) => {
+            if exit_status.success() {
+                Ok(())
+            } else {
+                bail!("{name:?}: Files are not formatted correctly.");
+            }
+        }
+        Err(e) => {
+            if e.kind() == std::io::ErrorKind::NotFound {
+                eprintln!(
+                    "{YELLOW}Formatter not found: {name:?}. Skipping associated files.{YELLOW:#}"
+                );
+                Ok(())
+            } else {
+                Err(e).with_context(|| format!("Error occurred while running {name:?}"))
+            }
+        }
+    }
+}
+
+fn format_fish(args: &FormatArgs) -> Result<()> {
+    let mut fish_paths = files_with_extension(&args.paths, "fish")?;
+    if args.all {
+        let workspace_root = fish_build_helper::workspace_root();
+        let fish_formatting_dirs = ["benchmarks", "build_tools", "etc", "share"];
+        fish_paths.extend(files_with_extension(
+            fish_formatting_dirs
+                .iter()
+                .map(|dir_name| workspace_root.join(dir_name)),
+            "fish",
+        )?);
+    };
+    if fish_paths.is_empty() {
+        return Ok(());
+    }
+    // TODO: make `fish_indent` available as a Rust library function, to avoid needing a
+    // `fish_indent` binary in `$PATH`.
+    let mut formatter = Command::new("fish_indent");
+    if args.check {
+        formatter.arg("--check");
+    } else {
+        formatter.arg("-w");
+    }
+    formatter.arg("--");
+    formatter.args(fish_paths);
+    run_formatter(&mut formatter, "fish_indent")
+}
+
+fn format_fluent(args: &FormatArgs) -> Result<()> {
+    println!("=== Running {GREEN}Fluent FTL formatter (built-in){GREEN:#}");
+    let mut ftl_files = files_with_extension(&args.paths, "ftl")?;
+    if args.all {
+        ftl_files.extend(files_with_extension([fish_build_helper::ftl_dir()], "ftl")?);
+    }
+    let mode = if args.check {
+        fluent_ftl_tools::format::FormattingMode::Check
+    } else {
+        fluent_ftl_tools::format::FormattingMode::Rewrite
+    };
+    let errors = ftl_files
+        .iter()
+        .filter_map(|path| fluent_ftl_tools::format::format_path(path, mode).err())
+        .collect::<Vec<anyhow::Error>>();
+    if !errors.is_empty() {
+        let mut error_message = String::from("Found these errors:\n");
+        for e in errors {
+            error_message.push_str(&format!("{e}\n"));
+        }
+        bail!("{error_message}");
+    }
+    Ok(())
+}
+
+fn format_python(args: &FormatArgs) -> Result<()> {
+    let mut formatter = Command::new("ruff");
+    formatter.arg("format");
+    if args.check {
+        formatter.arg("--check");
+    }
+    let mut python_files = files_with_extension(&args.paths, "py")?;
+
+    if args.all {
+        python_files.push(fish_build_helper::workspace_root().to_owned());
+    };
+    if python_files.is_empty() {
+        return Ok(());
+    }
+    formatter.args(python_files);
+    run_formatter(&mut formatter, "ruff format")
+}
+
+fn format_rust(args: &FormatArgs) -> Result<()> {
+    let rustfmt_status = Command::new("cargo")
+        .arg("fmt")
+        .arg("--version")
+        .stdout(Stdio::null())
+        .status()
+        .context("Failed to run cargo")?;
+    if !rustfmt_status.success() {
+        eprintln!(
+            "{YELLOW}Please install \"rustfmt\" to format Rust, e.g. via:\n\
+            rustup component add rustfmt{YELLOW:#}"
+        );
+        return Ok(());
+    }
+    if args.all {
+        let mut formatter = Command::new("cargo");
+        formatter.arg("fmt");
+        formatter.arg("--all");
+        if args.check {
+            formatter.arg("--check");
+        }
+        run_formatter(&mut formatter, "cargo fmt")?;
+    }
+    let rust_files = files_with_extension(&args.paths, "rs")?;
+    if rust_files.is_empty() {
+        return Ok(());
+    }
+    let mut formatter = Command::new("rustfmt");
+    if args.check {
+        formatter.arg("--check");
+        formatter.arg("--files-with-diff");
+    }
+    formatter.args(rust_files);
+    run_formatter(&mut formatter, "rustfmt")
+}

crates/xtask/src/gettext.rs

@@ -0,0 +1,552 @@
+use crate::{CARGO, CommandExt, files_with_extension};
+use anyhow::{Context as _, Result, bail};
+use clap::{Args, Subcommand};
+use fish_build_helper::po_dir;
+use pcre2::bytes::Regex;
+use std::{
+    fs::OpenOptions,
+    io::{Write as _, stdout},
+    path::{Path, PathBuf},
+    process::Command,
+    thread::spawn,
+};
+
+#[derive(Args)]
+pub struct GettextArgs {
+    /// Path to the directory into which the messages from the Rust sources have been extracted.
+    /// If this is not specified, fish will be compiled with the `gettext-extract` feature to
+    /// obtain the messages.
+    #[arg(long)]
+    rust_extraction_dir: Option<PathBuf>,
+
+    #[command(subcommand)]
+    task: Task,
+}
+
+#[derive(Subcommand)]
+enum Task {
+    /// Check whether the PO files are up to date.
+    /// Prints a diff and exits non-zero if they are outdated.
+    /// Considers all our PO files by default, also allows explicitly specifying which files to
+    /// consider.
+    Check { paths: Vec<PathBuf> },
+    /// Add a PO file for a new language.
+    New {
+        /// An ISO 639-1 language identifier (ISO 639-2 if the former does not exits),
+        /// optionally followed by and underscore and an ISO 3166-1 country code to specify the variant,
+        /// e.g. `de` or `pt_BR`.
+        language: String,
+    },
+    /// Update PO files.
+    /// This will delete entries for msgids which are no longer used in the sources and introduce
+    /// new, untranslated entries for messages which do not have an entry yet.
+    /// This tool should run every time a change to the messages localized via gettext occurs,
+    /// including fish script files, where many strings are implicitly localized.
+    /// Considers all our PO files by default, also allows explicitly specifying which files to
+    /// consider.
+    Update { paths: Vec<PathBuf> },
+}
+
+fn get_po_paths<P: AsRef<Path>>(specified_paths: &[P]) -> Result<Vec<PathBuf>> {
+    let extension = "po";
+    if specified_paths.is_empty() {
+        files_with_extension([po_dir()], extension)
+    } else {
+        files_with_extension(specified_paths, extension)
+    }
+}
+
+fn update_po_file<P: AsRef<Path>, Q: AsRef<Path>>(file_to_update: P, template: Q) -> Result<()> {
+    Command::new("msgmerge")
+        .args([
+            "--no-wrap",
+            "--update",
+            "--no-fuzzy-matching",
+            "--backup=none",
+            "--quiet",
+        ])
+        .arg(file_to_update.as_ref())
+        .arg(template.as_ref())
+        .run()?;
+    let msgattrib_output_file = fish_tempfile::new_file().context("Failed to create temp file")?;
+    Command::new("msgattrib")
+        .args(["--no-wrap", "--no-obsolete"])
+        .arg("-o")
+        .arg(msgattrib_output_file.path())
+        .arg(file_to_update.as_ref())
+        .run()?;
+    crate::copy_file(msgattrib_output_file.path(), file_to_update.as_ref())?;
+    Ok(())
+}
+
+pub fn gettext(args: GettextArgs) -> Result<()> {
+    let template = match args.rust_extraction_dir {
+        Some(dir) => template::Template::new(dir)?,
+        None => {
+            let temp_dir = fish_tempfile::new_dir().context("Failed to create temp file")?;
+            Command::new(CARGO)
+                .args([
+                    "check",
+                    "--workspace",
+                    "--all-targets",
+                    "--features=gettext-extract",
+                ])
+                .env("FISH_GETTEXT_EXTRACTION_DIR", temp_dir.path())
+                .run()?;
+            template::Template::new(temp_dir.path())?
+        }
+    };
+    let mut template_file = fish_tempfile::new_file().context("Failed to create temp file")?;
+    template_file
+        .get_mut()
+        .write_all(template.serialize())
+        .with_context(|| format!("Failed to write to temp file {:?}", template_file.path()))?;
+    template_file
+        .get_mut()
+        .flush()
+        .with_context(|| format!("Failed to flush temporary file {:?}", template_file.path()))?;
+
+    match args.task {
+        Task::Check { paths } => {
+            let mut thread_handles = vec![];
+            for path in get_po_paths(&paths)? {
+                let template_path_buf = template_file.path().to_owned();
+                let handle = spawn(move || -> Result<Option<Vec<u8>>> {
+                    let tmp_copy =
+                        fish_tempfile::new_file().context("Failed to create temp file")?;
+                    crate::copy_file(&path, tmp_copy.path())?;
+                    update_po_file(tmp_copy.path(), template_path_buf)?;
+                    let diff_output = Command::new("diff")
+                        .arg("-u")
+                        .arg(&path)
+                        .arg(tmp_copy.path())
+                        .output()
+                        .context("Failed to run diff")?;
+                    if diff_output.status.success() {
+                        Ok(None)
+                    } else {
+                        Ok(Some(diff_output.stdout))
+                    }
+                });
+                thread_handles.push(handle);
+            }
+            let mut found_diff = false;
+            let mut error = None;
+            for handle in thread_handles {
+                // SAFETY: `handle.join()` only returns `Err` if the thread panicked.
+                // Our threads should not panic, and if they do, it's OK to deal with the unexpected
+                // behavior by panicking here as well.
+                match handle.join().unwrap() {
+                    Ok(None) => {}
+                    Ok(Some(diff)) => {
+                        found_diff = true;
+                        stdout()
+                            .write_all(&diff)
+                            .context("Could not write to stdout")?;
+                    }
+                    Err(e) => {
+                        error = Some(e);
+                    }
+                }
+            }
+            if let Some(e) = error {
+                return Err(e);
+            }
+            if found_diff {
+                bail!(
+                    "Not all PO files are up to date.\n\
+                     Run `cargo xtask gettext update` to bring them up to date automatically.\
+                    "
+                );
+            }
+            Ok(())
+        }
+        Task::New { language } => {
+            let language_regex = Regex::new("^[a-z]{2,3}(_[A-Z]{2})?$").unwrap();
+            if !language_regex.is_match(language.as_bytes()).unwrap() {
+                bail!(
+                    "The language name '{language}' does not match the expected format.\n\
+                    It needs to be a two-letter ISO 639-1 language code, \
+                    or a three-letter ISO 639-2 language code \
+                    if no ISO 639-1 code exists for the language.\n\
+                    Optionally, the language code can be followed be an underscore \
+                    followed by an ISO 3166-1 country code to indicate a regional variant.\n\
+                    Check the existing file names in {:?} for examples.",
+                    po_dir()
+                );
+            }
+            // TODO (MSRV>=1.91): use with_added_extension instead of with_extension
+            let po_path = po_dir().join(&language).with_extension("po");
+            let mut new_po_file = OpenOptions::new()
+                .create_new(true)
+                .write(true)
+                .open(&po_path)
+                .with_context(|| format!("Failed to create file at {po_path:?}"))?;
+            let mut header = String::new();
+            let line_prefix = "# fish-note-sections: ";
+            let lines = [
+                "Translations are divided into sections, each starting with a fish-section-* pseudo-message.",
+                "The first few sections are more important.",
+                "Ignore the tier3 sections unless you have a lot of time.",
+            ];
+            for line in lines {
+                use std::fmt::Write as _;
+                let _ = writeln!(header, "{line_prefix}{line}");
+            }
+            new_po_file
+                .write_all(header.as_bytes())
+                .with_context(|| format!("Failed to write to {po_path:?}"))?;
+            new_po_file
+                .write_all(template.serialize())
+                .with_context(|| format!("Failed to write to {po_path:?}"))?;
+            Ok(())
+        }
+        Task::Update { paths } => {
+            let mut thread_handles = vec![];
+            for path in get_po_paths(&paths)? {
+                let template_path_buf = template_file.path().to_owned();
+                let handle =
+                    spawn(move || -> Result<()> { update_po_file(path, template_path_buf) });
+                thread_handles.push(handle);
+            }
+            let mut error = None;
+            for handle in thread_handles {
+                // SAFETY: `handle.join()` only returns `Err` if the thread panicked.
+                // Our threads should not panic, and if they do, it's OK to deal with the unexpected
+                // behavior by panicking here as well.
+                if let Err(e) = handle.join().unwrap() {
+                    error = Some(e);
+                }
+            }
+            if let Some(e) = error {
+                return Err(e);
+            }
+            Ok(())
+        }
+    }
+}
+
+mod template {
+    use crate::{CommandExt as _, files_with_extension};
+    use anyhow::{Context as _, Result, bail};
+    use fish_build_helper::workspace_root;
+    use fish_common::{UnescapeFlags, unescape_string};
+    use fish_widestring::{str2wcstring, wcs2bytes};
+    use pcre2::bytes::Regex;
+    use std::{
+        collections::{HashMap, HashSet},
+        fmt::Display,
+        fs::OpenOptions,
+        io::Read as _,
+        path::Path,
+        process::Command,
+        sync::LazyLock,
+    };
+
+    // Gettext tools require this header to know which encoding is used.
+    const MINIMAL_HEADER: &str = r#"msgid ""
+msgstr "Content-Type: text/plain; charset=UTF-8\n"
+
+"#;
+
+    #[derive(PartialEq, Eq, PartialOrd, Ord, Clone, Copy, Hash)]
+    enum LocalizationTier {
+        Tier1,
+        Tier2,
+        Tier3,
+    }
+
+    impl TryFrom<&str> for LocalizationTier {
+        type Error = ();
+
+        fn try_from(value: &str) -> Result<Self, Self::Error> {
+            match value {
+                "tier1" => Ok(Self::Tier1),
+                "tier2" => Ok(Self::Tier2),
+                "tier3" => Ok(Self::Tier3),
+                _ => Err(()),
+            }
+        }
+    }
+
+    impl Display for LocalizationTier {
+        fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+            f.write_str(match self {
+                Self::Tier1 => "tier1",
+                Self::Tier2 => "tier2",
+                Self::Tier3 => "tier3",
+            })
+        }
+    }
+
+    #[derive(Default)]
+    struct FishScriptMessages {
+        explicit: HashSet<String>,
+        implicit: HashSet<String>,
+    }
+
+    pub struct Template {
+        content: Vec<u8>,
+    }
+
+    impl Template {
+        pub fn serialize(&self) -> &[u8] {
+            &self.content
+        }
+
+        /// Create a gettext template.
+        /// `rust_extraction_dir` must be the path to a directory which contains the messages
+        /// extracted from the Rust sources.
+        pub fn new<P: AsRef<Path>>(rust_extraction_dir: P) -> Result<Self> {
+            let mut template = Self {
+                content: Vec::from(MINIMAL_HEADER.as_bytes()),
+            };
+            template.add_rust_messages(rust_extraction_dir)?;
+            template.add_fish_script_messages()?;
+            // TODO: keep internal set of msgids to avoid having to run msguniq. requires parsing
+            // gettext-extraction output
+            let msguniq_output = Command::new("msguniq")
+                .args(["--no-wrap"])
+                .run_with_stdio(&template.content)?;
+            Ok(Template {
+                content: msguniq_output,
+            })
+        }
+
+        /// Expects `extraction_dir` to contain only files whose content are single PO entries which can be
+        /// concatenated into a valid PO file.
+        /// If this is the case, the messages are de-duplicated and sorted by `msguniq`.
+        /// The result is appended to `template`, with a leading section marker.
+        /// On failure, the process aborts.
+        fn add_rust_messages<P: AsRef<Path>>(&mut self, extraction_dir: P) -> Result<()> {
+            let extraction_dir = extraction_dir.as_ref();
+            let mut concatenated_content = Vec::from(MINIMAL_HEADER.as_bytes());
+
+            // Concatenate the content of all files in `extraction_dir` into `concatenated_content`.
+            for entry_result in extraction_dir
+                .read_dir()
+                .with_context(|| format!("Failed to read directory {extraction_dir:?}"))?
+            {
+                let entry = entry_result
+                    .with_context(|| format!("Failed to get entry in {extraction_dir:?}"))?;
+                let entry_path = entry.path();
+                if !entry
+                    .file_type()
+                    .with_context(|| format!("Failed to get file type of {entry_path:?}"))?
+                    .is_file()
+                {
+                    bail!("Entry in {extraction_dir:?} is not a regular file");
+                }
+                let mut file = OpenOptions::new()
+                    .read(true)
+                    .open(&entry_path)
+                    .with_context(|| format!("Failed to open file {entry_path:?}"))?;
+                file.read_to_end(&mut concatenated_content)
+                    .with_context(|| format!("Failed to read file {entry_path:?}"))?;
+            }
+
+            // Get rid of duplicates and sort.
+            let msguniq_output = Command::new("msguniq")
+                .args(["--no-wrap", "--sort-output"])
+                .env("LC_ALL", "C.UTF-8")
+                .run_with_stdio(&concatenated_content)?;
+            // The Header entry needs to be removed again,
+            // because it is added outside of this function.
+            let expected_prefix = MINIMAL_HEADER.as_bytes();
+            let actual_prefix = &msguniq_output[..expected_prefix.len()];
+            if expected_prefix != actual_prefix {
+                bail!(
+                    "Prefix of msguniq output does not match expected header.\nExpected bytes:\n{expected_prefix:02x?}\nActual bytes:\n{actual_prefix:02x?}"
+                );
+            }
+            self.mark_section("tier1-from-rust");
+            self.content
+                .extend_from_slice(&msguniq_output[expected_prefix.len()..]);
+            self.content.push(b'\n');
+            Ok(())
+        }
+
+        fn mark_section(&mut self, section_name: &str) {
+            self.content
+                .extend_from_slice("msgid \"fish-section-".as_bytes());
+            self.content.extend_from_slice(section_name.as_bytes());
+            self.content
+                .extend_from_slice("\"\nmsgstr \"\"\n\n".as_bytes());
+        }
+
+        fn append_messages(&mut self, msgids: &HashSet<String>) -> Result<()> {
+            let mut unescaped_msgids = HashSet::new();
+            for msgid in msgids {
+                let unescaped_wstring = unescape_string(
+                    &str2wcstring(msgid),
+                    fish_common::UnescapeStringStyle::Script(UnescapeFlags::default()),
+                )
+                .with_context(|| format!("Failed to unescape the following string:\n{msgid}"))?;
+                unescaped_msgids.insert(
+                    String::from_utf8(wcs2bytes(&unescaped_wstring))
+                        .context("Parsed msgid is not valid UTF-8")?,
+                );
+            }
+            let mut unescaped_msgids = Vec::from_iter(unescaped_msgids);
+            unescaped_msgids.sort();
+            for msgid in &unescaped_msgids {
+                self.content
+                    .extend_from_slice(format_msgid_for_po(msgid).as_bytes());
+            }
+            Ok(())
+        }
+
+        fn add_script_tier(
+            &mut self,
+            tier: LocalizationTier,
+            messages: FishScriptMessages,
+        ) -> Result<()> {
+            if !messages.explicit.is_empty() {
+                self.mark_section(&format!("{tier}-from-script-explicitly-added"));
+                self.append_messages(&messages.explicit)?;
+            }
+            if !messages.implicit.is_empty() {
+                self.mark_section(&format!("{tier}-from-script-implicitly-added"));
+                self.append_messages(&messages.implicit)?;
+            }
+            Ok(())
+        }
+
+        fn add_fish_script_messages(&mut self) -> Result<()> {
+            let share_dir = workspace_root().join("share");
+            let relevant_file_paths = files_with_extension(
+                [
+                    share_dir.join("config.fish"),
+                    share_dir.join("completions"),
+                    share_dir.join("functions"),
+                ],
+                "fish",
+            )?;
+            let mut extracted_messages = HashMap::new();
+            for path in relevant_file_paths {
+                extract_messages_from_fish_script(path, &mut extracted_messages)?;
+            }
+            let mut messages_sorted_by_tier: Vec<_> = extracted_messages.into_iter().collect();
+            messages_sorted_by_tier.sort_by_key(|(tier, _)| *tier);
+            for (tier, messages) in messages_sorted_by_tier {
+                self.add_script_tier(tier, messages)?;
+            }
+            Ok(())
+        }
+    }
+
+    fn find_localization_tier<P: AsRef<Path>>(
+        input: &str,
+        path: P,
+    ) -> Result<Option<LocalizationTier>> {
+        static L10N_ANNOTATION: LazyLock<Regex> = LazyLock::new(|| {
+            Regex::new(r"(?:^|\n)# localization: (?<annotation_value>.*)\n").unwrap()
+        });
+        if let Some(annotation) = L10N_ANNOTATION.captures(input.as_bytes()).unwrap() {
+            // SAFETY: `tier` is the name of a capture group in the regex whose captures we are looking
+            // at. The capture is done on the bytes of an UTF-8 encoded string, so the result will also
+            // be UTF-8 encoded, and the sub-slice we are looking at will start and end at codepoint
+            // boundaries.
+            let annotation_value =
+                std::str::from_utf8(annotation.name("annotation_value").unwrap().as_bytes())
+                    .unwrap();
+            if let Ok(tier) = LocalizationTier::try_from(annotation_value) {
+                return Ok(Some(tier));
+            }
+            if annotation_value.starts_with("skip") {
+                return Ok(None);
+            }
+            bail!(
+                "Unexpected localization annotation in file {:?}: {annotation_value}",
+                path.as_ref()
+            );
+        }
+        let dirname = path
+            .as_ref()
+            .parent()
+            .with_context(|| {
+                format!(
+                    "Tried to get the parent of a path which does not have a parent: {:?}",
+                    path.as_ref()
+                )
+            })?
+            .file_name()
+            .with_context(|| {
+                format!(
+                    "The parent of {:?} does not have a filename component",
+                    path.as_ref()
+                )
+            })?;
+        let command_name = path
+            .as_ref()
+            .file_stem()
+            .with_context(|| format!("The path {:?} does not have a file stem", path.as_ref()))?;
+        if dirname == "functions"
+            && command_name
+                .to_str()
+                .is_some_and(|name| name.starts_with("fish_"))
+        {
+            return Ok(Some(LocalizationTier::Tier1));
+        }
+        if dirname != "completions" {
+            bail!(
+                "Missing localization tier for function file {:?}",
+                path.as_ref()
+            );
+        }
+        // TODO (MSRV>=1.91): use with_added_extension instead of with_extension
+        let doc_path = workspace_root()
+            .join("doc_src")
+            .join("cmds")
+            .join(command_name)
+            .with_extension("rst");
+        let doc_path_exists = std::fs::exists(&doc_path)
+            .with_context(|| format!("Failed to check whether a file exists at {doc_path:?}"))?;
+        Ok(Some(if doc_path_exists {
+            LocalizationTier::Tier1
+        } else {
+            LocalizationTier::Tier3
+        }))
+    }
+
+    fn extract_messages_from_fish_script<P: AsRef<Path>>(
+        path: P,
+        extracted_messages: &mut HashMap<LocalizationTier, FishScriptMessages>,
+    ) -> Result<()> {
+        let path = path.as_ref();
+        let file_content = std::fs::read_to_string(path)
+            .with_context(|| format!("Failed to read from {path:?}"))?;
+        let Some(tier) = find_localization_tier(&file_content, path)? else {
+            return Ok(());
+        };
+        // TODO: use proper parser instead of regex
+        static EXPLICIT_MESSAGE: LazyLock<Regex> =
+            LazyLock::new(|| Regex::new(r#"\( *_ (?<message>(['"]).+?(?<!\\)\2) *\)"#).unwrap());
+        static IMPLICIT_MESSAGE: LazyLock<Regex> = LazyLock::new(|| {
+            Regex::new(r#"(?:^|\n)(?:\s|and |or )*(?:complete|function).*? (?:-d|--description) (?<message>(['"]).+?(?<!\\)\2)"#).unwrap()
+        });
+        let messages_at_tier = extracted_messages.entry(tier).or_default();
+        for message in EXPLICIT_MESSAGE.captures_iter(file_content.as_bytes()) {
+            let message =
+                std::str::from_utf8(message.unwrap().name("message").unwrap().as_bytes()).unwrap();
+            messages_at_tier.explicit.insert(message.to_owned());
+        }
+        for message in IMPLICIT_MESSAGE.captures_iter(file_content.as_bytes()) {
+            let message =
+                std::str::from_utf8(message.unwrap().name("message").unwrap().as_bytes()).unwrap();
+            messages_at_tier.implicit.insert(message.to_owned());
+        }
+        Ok(())
+    }
+
+    fn format_msgid_for_po(msgid: &str) -> String {
+        let escaped_msgid = msgid.replace("\\", "\\\\").replace("\"", "\\\"");
+        format!(
+            "\
+            msgid \"{escaped_msgid}\"\n\
+            msgstr \"\"\n\
+            \n\
+            "
+        )
+    }
+}

crates/xtask/src/lib.rs

@@ -1,28 +1,102 @@
-use std::{ffi::OsStr, process::Command};
+use std::{
+    ffi::OsStr,
+    io::Write,
+    path::{Path, PathBuf},
+    process::{Command, Stdio},
+};
+
+use anyhow::{Context, Result, bail};
+use walkdir::WalkDir;
+
+pub mod fluent;
+pub mod format;
+pub mod gettext;
+pub mod shellcheck;
 
 pub trait CommandExt {
-    fn run_or_panic(&mut self);
+    fn run(&mut self) -> Result<()>;
+    fn run_with_stdio(&mut self, stdin: &[u8]) -> Result<Vec<u8>>;
 }
 
 impl CommandExt for Command {
-    fn run_or_panic(&mut self) {
-        match self.status() {
-            Ok(exit_status) => {
-                if !exit_status.success() {
-                    panic!("Command did not run successfully: {:?}", self.get_program())
-                }
-            }
-            Err(err) => {
-                panic!("Failed to run command: {err}");
-            }
+    fn run(&mut self) -> Result<()> {
+        if !self
+            .status()
+            .with_context(|| format!("Failed to run {:?}", self.get_program()))?
+            .success()
+        {
+            bail!("Command did not run successfully: {:?}", self.get_program())
         }
+        Ok(())
+    }
+
+    fn run_with_stdio(&mut self, stdin: &[u8]) -> Result<Vec<u8>> {
+        let command_name = self.get_program().to_owned();
+        let mut child = self
+            .stdin(Stdio::piped())
+            .stdout(Stdio::piped())
+            .spawn()
+            .with_context(|| format!("Failed to run {command_name:?}"))?;
+        child
+            .stdin
+            .take()
+            .unwrap()
+            .write_all(stdin)
+            .with_context(|| format!("Failed to write to stdin of {command_name:?}"))?;
+        let command_output = child
+            .wait_with_output()
+            .with_context(|| format!("Failed to read stdout of {command_name:?}"))?;
+        if !command_output.status.success() {
+            bail!("{command_name:?} failed");
+        }
+        Ok(command_output.stdout)
     }
 }
 
-pub fn cargo<I, S>(cargo_args: I)
+pub const CARGO: &str = env!("CARGO");
+
+pub fn cargo<I, S>(cargo_args: I) -> Result<()>
 where
     I: IntoIterator<Item = S>,
     S: AsRef<OsStr>,
 {
-    Command::new(env!("CARGO")).args(cargo_args).run_or_panic();
+    Command::new(CARGO).args(cargo_args).run()
+}
+
+fn get_matching_files<P: AsRef<Path>, I: IntoIterator<Item = P>, M: Fn(&Path) -> bool>(
+    all_paths: I,
+    matcher: M,
+) -> Result<Vec<PathBuf>> {
+    let mut matching_files = vec![];
+    for path in all_paths {
+        for dir_entry in WalkDir::new(path.as_ref()) {
+            let dir_entry = dir_entry
+                .with_context(|| format!("Failed to check paths at {:?}", path.as_ref()))?;
+            let path = dir_entry.path();
+            if dir_entry.file_type().is_file() && matcher(path) {
+                matching_files.push(path.to_owned());
+            }
+        }
+    }
+    Ok(matching_files)
+}
+
+fn files_with_extension<P: AsRef<Path>, I: IntoIterator<Item = P>>(
+    all_paths: I,
+    extension: &str,
+) -> Result<Vec<PathBuf>> {
+    let matcher = |p: &Path| p.extension().is_some_and(|e| e == extension);
+    get_matching_files(all_paths, matcher)
+}
+
+fn copy_file<P: AsRef<Path>, Q: AsRef<Path>>(from: P, to: Q) -> Result<()> {
+    std::fs::copy(&from, &to)
+        .with_context(|| {
+            format!(
+                "Failed to copy from {:?} to {:?}",
+                from.as_ref(),
+                to.as_ref()
+            )
+        })
+        .map(|_| ())
 }

crates/xtask/src/main.rs

@@ -1,7 +1,12 @@
-use clap::{Parser, Subcommand};
+use anyhow::{Context, Result};
+use clap::{CommandFactory, Parser, Subcommand};
+use clap_complete::CompleteEnv;
 use fish_build_helper::as_os_strs;
 use std::{path::PathBuf, process::Command};
-use xtask::{CommandExt as _, cargo};
+use xtask::{
+    CommandExt, cargo, fluent::FluentCommandArgs, format::FormatArgs, gettext::GettextArgs,
+    shellcheck::shellcheck,
+};
 
 #[derive(Parser)]
 #[command(
@@ -18,6 +23,13 @@ struct Cli {
 enum Task {
     /// Run various checks on the repo.
     Check,
+    /// Run Fluent-related tools.
+    #[command(subcommand)]
+    Fluent(FluentCommandArgs),
+    /// Format files or check if they are correctly formatted.
+    Format(FormatArgs),
+    /// Work on the gettext PO files.
+    Gettext(GettextArgs),
     /// Build HTML docs
     HtmlDocs {
         /// Path to a fish_indent executable. If none is specified, fish_indent will be built.
@@ -26,52 +38,86 @@ enum Task {
     },
     /// Build man pages
     ManPages,
+    /// Run ShellCheck on non-fish shell scripts
+    #[command(name = "shellcheck")]
+    ShellCheck,
 }
 
-fn main() {
+/// Only used to enable completion generation.
+/// [`clap_complete`] is not built to account for the situation we have here, where the CLI does not
+/// correspond to a top-level shell command.
+/// We work around this here by pretending that we are building a CLI for the `cargo` command, which
+/// only has the single subcommand `xtask`.
+/// These completions can then be combined with the regular cargo completions.
+#[derive(Parser)]
+#[command(name = "cargo")]
+struct FakeCargoWrapperForCompletion {
+    #[command(subcommand)]
+    xtask: FakeCliForCompletion,
+}
+
+#[derive(Subcommand)]
+enum FakeCliForCompletion {
+    /// Run fish's xtasks
+    #[command(subcommand)]
+    Xtask(Task),
+}
+
+fn main() -> Result<()> {
+    CompleteEnv::with_factory(FakeCargoWrapperForCompletion::command).complete();
+
     let cli = Cli::parse();
     match cli.task {
         Task::Check => run_checks(),
+        Task::Fluent(fluent_command_args) => xtask::fluent::fluent(fluent_command_args),
+        Task::Format(format_args) => xtask::format::format(format_args),
+        Task::Gettext(gettext_args) => xtask::gettext::gettext(gettext_args),
         Task::HtmlDocs { fish_indent } => build_html_docs(fish_indent),
         Task::ManPages => cargo(["build", "--package", "fish-build-man-pages"]),
+        Task::ShellCheck => shellcheck(),
     }
 }
 
-fn run_checks() {
+fn run_checks() -> Result<()> {
     let repo_root_dir = fish_build_helper::workspace_root();
     let check_script = repo_root_dir.join("build_tools").join("check.sh");
-    Command::new(check_script).run_or_panic();
+    Command::new(check_script).run()
 }
 
-fn build_html_docs(fish_indent: Option<PathBuf>) {
-    let fish_indent_path = fish_indent.unwrap_or_else(|| {
-        // Build fish_indent if no existing one is specified.
-        cargo([
-            "build",
-            "--bin",
-            "fish_indent",
-            "--profile",
-            "dev",
-            "--no-default-features",
-        ]);
-        fish_build_helper::fish_build_dir()
-            .join("debug")
-            .join("fish_indent")
-    });
+fn build_html_docs(fish_indent: Option<PathBuf>) -> Result<()> {
+    let fish_indent_path = match fish_indent {
+        Some(path) => path,
+        None => {
+            // Build fish_indent if no existing one is specified.
+            cargo([
+                "build",
+                "--bin",
+                "fish_indent",
+                "--profile",
+                "dev",
+                "--no-default-features",
+            ])?;
+            fish_build_helper::fish_build_dir()
+                .join("debug")
+                .join("fish_indent")
+        }
+    };
     // Set path so `sphinx-build` can find `fish_indent`.
     // Create tempdir to store symlink to fish_indent.
     // This is done to avoid adding other binaries to the PATH.
-    let tempdir = fish_tempfile::new_dir().unwrap();
+    let tempdir = fish_tempfile::new_dir().context("Failed to create tempdir")?;
     std::os::unix::fs::symlink(
-        std::fs::canonicalize(fish_indent_path).unwrap(),
+        std::fs::canonicalize(&fish_indent_path).with_context(|| {
+            format!("Failed to canonicalize path to `fish_indent`: {fish_indent_path:?}")
+        })?,
         tempdir.path().join("fish_indent"),
     )
-    .unwrap();
-    let new_path = format!(
-        "{}:{}",
-        tempdir.path().to_str().unwrap(),
-        fish_build_helper::env_var("PATH").unwrap()
-    );
+    .context("Failed to create symlink for fish_indent")?;
+    let mut new_path = tempdir.path().as_os_str().to_owned();
+    if let Some(current_path) = std::env::var_os("PATH") {
+        new_path.push(":");
+        new_path.push(current_path);
+    }
     let doc_src_dir = fish_build_helper::workspace_root().join("doc_src");
     let doctrees_dir = fish_build_helper::fish_doc_dir().join(".doctrees-html");
     let html_dir = fish_build_helper::fish_doc_dir().join("html");
@@ -91,5 +137,5 @@ fn build_html_docs(fish_indent: Option<PathBuf>) {
     Command::new(option_env!("FISH_SPHINX").unwrap_or("sphinx-build"))
         .env("PATH", new_path)
         .args(args)
-        .run_or_panic();
+        .run()
 }

crates/xtask/src/shellcheck.rs

@@ -0,0 +1,52 @@
+use anyhow::{Context, Result};
+use fish_build_helper::workspace_root;
+use ignore::Walk;
+use pcre2::bytes::Regex;
+use std::{
+    fs::File,
+    io::{BufRead, BufReader},
+    path::{Path, PathBuf},
+    process::Command,
+    sync::LazyLock,
+};
+
+use crate::CommandExt;
+
+pub fn shellcheck() -> Result<()> {
+    Command::new("shellcheck")
+        .args(files_to_check()?)
+        .current_dir(workspace_root())
+        .run()
+}
+
+fn is_shell_script<P: AsRef<Path>>(path: P) -> Result<bool> {
+    let file = File::open(&path).with_context(|| format!("Failed to open {:?}", path.as_ref()))?;
+    let mut first_line = String::new();
+    let Ok(_) = BufReader::new(file).read_line(&mut first_line) else {
+        return Ok(false);
+    };
+    static SHEBANG_REGEX: LazyLock<Regex> = LazyLock::new(|| Regex::new("^#!.*[^i]sh").unwrap());
+    Ok(SHEBANG_REGEX
+        .is_match(first_line.trim().as_bytes())
+        .unwrap())
+}
+
+fn files_to_check() -> Result<Vec<PathBuf>> {
+    let mut files = vec![];
+    for dir_entry in Walk::new(workspace_root()) {
+        let dir_entry = dir_entry.context("Error traversing workspace")?;
+        if !dir_entry
+            .file_type()
+            .with_context(|| format!("Failed to determine file type of {dir_entry:?}"))?
+            .is_file()
+        {
+            continue;
+        }
+        let path = dir_entry.into_path();
+        if !is_shell_script(&path)? {
+            continue;
+        }
+        files.push(path);
+    }
+    Ok(files)
+}

deny.toml

@@ -1,24 +1,24 @@
 [licenses]
 # We want really high confidence when inferring licenses from text
 confidence-threshold = 0.93
-unused-allowed-license = "allow" # don't warn for unused licenses in this list
+unused-allowed-license = "allow"  # don't warn for unused licenses in this list
 allow = [
-    "BSD-2-Clause",
-    "BSD-3-Clause",
-    "BSL-1.0",
-    "CC0-1.0",
-    "GPL-2.0",
-    "GPL-2.0-only",
-    "ISC",
-    "LGPL-2.0",
-    "LGPL-2.0-or-later",
-    "MIT",
-    "MPL-2.0",
-    "PSF-2.0",
-    "Unicode-DFS-2016",
-    "Unicode-3.0",
-    "WTFPL",
-    "Zlib",
+  "BSD-2-Clause",
+  "BSD-3-Clause",
+  "BSL-1.0",
+  "CC0-1.0",
+  "GPL-2.0",
+  "GPL-2.0-only",
+  "ISC",
+  "LGPL-2.0",
+  "LGPL-2.0-or-later",
+  "MIT",
+  "MPL-2.0",
+  "PSF-2.0",
+  "Unicode-DFS-2016",
+  "Unicode-3.0",
+  "WTFPL",
+  "Zlib",
 ]
 
 [sources.allow-org]

doc_src/cmds/argparse.rst

@@ -46,7 +46,7 @@ The following ``argparse`` options are available. They must appear before all *O
     In contrast, if the known option comes first (and does not take any arguments), the known option will be recognised (e.g. ``argparse --move-unknown h -- -ho`` *will* set ``$_flag_h`` to ``-h``)
 
 **-i** or **--ignore-unknown**
-    Deprecated. This is like **--move-unknown**, except that unknown options and their arguments are kept in ``$argv`` and not moved to ``$argv_opts``. Unlike **--move-unknown**, this option makes it impossible to distinguish between an unknown option and non-option argument that starts with a ``-`` (since any ``--`` seperator in ``$argv`` will be removed).
+    Deprecated. This is like **--move-unknown**, except that unknown options and their arguments are kept in ``$argv`` and not moved to ``$argv_opts``. Unlike **--move-unknown**, this option makes it impossible to distinguish between an unknown option and non-option argument that starts with a ``-`` (since any ``--`` separator in ``$argv`` will be removed).
 
 **-S** or **--strict-longopts**
     This makes the parsing of long options more strict. In particular, *without* this flag, if ``long`` is a known long option flag, ``--long`` and ``--long=<value>`` can be abbreviated as:
@@ -221,7 +221,7 @@ These variables are passed to the function as local exported variables.
 
 The script should write any error messages to stdout, not stderr. It should return a status of zero if the flag value is valid otherwise a non-zero status to indicate it is invalid.
 
-Fish ships with a ``_validate_int`` function that accepts a ``--min`` and ``--max`` flag. Let's say your command accepts a ``-m`` or ``--max`` flag and the minimum allowable value is zero and the maximum is 5. You would define the option like this: ``m/max=!_validate_int --min 0 --max 5``. The default if you call ``_validate_int`` without those flags is to check that the value is a valid integer with no limits on the min or max value allowed.
+fish ships with a ``_validate_int`` function that accepts a ``--min`` and ``--max`` flag. Let's say your command accepts a ``-m`` or ``--max`` flag and the minimum allowable value is zero and the maximum is 5. You would define the option like this: ``m/max=!_validate_int --min 0 --max 5``. The default if you call ``_validate_int`` without those flags is to check that the value is a valid integer with no limits on the min or max value allowed.
 
 Here are some examples of flag validations::
 
@@ -251,7 +251,7 @@ Some *OPTION_SPEC* examples:
 
 - ``n/name=?`` means that both ``-n`` and ``--name`` are valid. It accepts an optional value and can be used at most once. If the flag is seen then ``_flag_n`` and ``_flag_name`` will be set with the value associated with the flag if one was provided else it will be set with no values.
 
-- ``n/name=*`` is similar, but the flag can be used more than once. If the flag is seen then ``_flag_n`` and ``_flag_name`` will be set with the values associated with each occurence. Each value will be the value given to the option, or the empty string if no value was given.
+- ``n/name=*`` is similar, but the flag can be used more than once. If the flag is seen then ``_flag_n`` and ``_flag_name`` will be set with the values associated with each occurrence. Each value will be the value given to the option, or the empty string if no value was given.
 
 - ``name=+`` means that only ``--name`` is valid. It requires a value and can be used more than once. If the flag is seen then ``_flag_name`` will be set with the values associated with each occurrence.
 

doc_src/cmds/cd.rst

@@ -6,7 +6,7 @@ Synopsis
 
 .. synopsis::
 
-    cd [DIRECTORY]
+    cd [( -L | --no-dereference ) | ( -P | --dereference )] [DIRECTORY]
 
 Description
 -----------
@@ -18,14 +18,25 @@ Description
 
 ``cd`` changes the current working directory.
 
+The :envvar:`PWD` environment variable is updated with the new working directory, and the previous directory
+is added to the :ref:`directory history <directory-history>`.
+
 If *DIRECTORY* is given, it will become the new directory. If no parameter is given, the :envvar:`HOME` environment variable will be used.
 
 If *DIRECTORY* is a relative path, all the paths in the :envvar:`CDPATH` will be tried as prefixes for it, in addition to :envvar:`PWD`.
 It is recommended to keep **.** as the first element of :envvar:`CDPATH`, or :envvar:`PWD` will be tried last.
 
-Fish will also try to change directory if given a command that looks like a directory (starting with **.**, **/** or **~**, or ending with **/**), without explicitly requiring **cd**.
+The new directory name is partially resolved to remove redundant segments (``.`` or ``..``).
 
-Fish also ships a wrapper function around the builtin **cd** that understands ``cd -`` as changing to the previous directory.
+``cd`` defaults to treating symbolic links as real directories, and not resolving them to their underlying
+targets. The ``$PWD`` :ref:`special variable <variables-special>` variable will contain the path that was
+supplied. This default behaviour can be enforced with the ``-L`` or ``--no-dereference`` option.
+
+The ``-P`` or ``--dereference`` option resolves all symbolic links first. This was the default in fish versions before 3.0.0.
+
+fish will also try to change directory if given a command that looks like a directory (starting with **.**, **/** or **~**, or ending with **/**), without explicitly requiring **cd**.
+
+fish also ships a wrapper function around the builtin **cd** that understands ``cd -`` as changing to the previous directory.
 See also :doc:`prevd <prevd>`.
 This wrapper function maintains a history of the 25 most recently visited directories in the ``$dirprev`` and ``$dirnext`` global variables.
 If you make those universal variables your **cd** history is shared among all fish instances.
@@ -45,6 +56,9 @@ Examples
     cd /usr/src/fish-shell
     # changes the working directory to /usr/src/fish-shell
 
+    cd -P /tmp/link
+    # resolves /tmp/link to its target before recording the directory
+
 See Also
 --------
 

doc_src/cmds/complete.rst

@@ -81,7 +81,7 @@ The following options are available:
 **-h** or **--help**
     Displays help about using this command.
 
-Command-specific tab-completions in ``fish`` are based on the notion of options and arguments. An option is a parameter which begins with a hyphen, such as ``-h``, ``-help`` or ``--help``. Arguments are parameters that do not begin with a hyphen. Fish recognizes three styles of options, the same styles as the GNU getopt library. These styles are:
+Command-specific tab-completions in ``fish`` are based on the notion of options and arguments. An option is a parameter which begins with a hyphen, such as ``-h``, ``-help`` or ``--help``. Arguments are parameters that do not begin with a hyphen. fish recognizes three styles of options, the same styles as the GNU getopt library. These styles are:
 
 - Short options, like ``-a``. Short options are a single character long, are preceded by a single hyphen and can be grouped together (like ``-la``, which is equivalent to ``-l -a``). Option arguments may be specified by appending the option with the value (``-w32``), or, if ``--require-parameter`` is given, in the following parameter (``-w 32``).
 

doc_src/cmds/fish_breakpoint_prompt.rst

@@ -34,6 +34,6 @@ A simple prompt that is a simplified version of the default debugging prompt::
         set -l function (status current-function)
         set -l line (status current-line-number)
         set -l prompt "$function:$line >"
-        echo -ns (set_color $fish_color_status) "BP $prompt" (set_color normal) ' '
+        echo -ns (set_color $fish_color_status) "BP $prompt" (set_color --reset) ' '
     end
 

doc_src/cmds/fish_command_not_found.rst

@@ -18,7 +18,7 @@ When fish tries to execute a command and can't find it, it invokes this function
 
 It can print a message to tell you about it, and it often also checks for a missing package that would include the command.
 
-Fish ships multiple handlers for various operating systems and chooses from them when this function is loaded,
+fish ships multiple handlers for various operating systems and chooses from them when this function is loaded,
 or you can define your own.
 
 It receives the full commandline as one argument per token, so $argv[1] contains the missing command.
@@ -50,25 +50,3 @@ Or the simple default handler::
     function fish_command_not_found
         __fish_default_command_not_found_handler $argv
     end
-
-Backwards compatibility
------------------------
-
-This command was introduced in fish 3.2.0. Previous versions of fish used the "fish_command_not_found" :ref:`event <event>` instead.
-
-To define a handler that works in older versions of fish as well, define it the old way::
-
-  function __fish_command_not_found_handler --on-event fish_command_not_found
-       echo COMMAND WAS NOT FOUND MY FRIEND $argv[1]
-  end
-
-in which case fish will define a ``fish_command_not_found`` that calls it,
-or define a wrapper::
-
-  function fish_command_not_found
-       echo "G'day mate, could not find your command: $argv"
-  end
-
-  function __fish_command_not_found_handler --on-event fish_command_not_found
-       fish_command_not_found $argv
-  end

doc_src/cmds/fish_config.rst

@@ -85,10 +85,10 @@ The format looks like this:
     fish_color_command 5c5cff
 
     [unknown]
-    fish_color_normal normal
+    fish_color_normal --reset
     fish_color_autosuggestion brblack
     fish_color_cancel -r
-    fish_color_command normal
+    fish_color_command --reset
 
 The comments provide name and background color to the web config tool.
 

doc_src/cmds/fish_delta.rst

@@ -30,13 +30,13 @@ Options
 The following options are available:
 
 **-f** or **--no-functions**
-    Stops checking functions
+    Skips checking functions
 
 **-c** or **--no-completions**
-    Stops checking completions
+    Skips checking completions
 
 **-C** or **--no-config**
-    Stops checking configuration files like config.fish or snippets in the conf.d directories.
+    Skips checking configuration files like config.fish or snippets in the conf.d directories.
 
 **-d** or **--no-diff**
     Removes the diff display (this happens automatically if ``diff`` can't be found)

doc_src/cmds/fish_greeting.rst

@@ -22,6 +22,8 @@ When an interactive fish starts, it executes fish_greeting and displays its outp
 
 The default fish_greeting is a function that prints a variable of the same name (``$fish_greeting``), so you can also just change that if you just want to change the text.
 
+If :envvar:`SHELL_WELCOME` is set, it is displayed after the greeting. This is a standard environment variable that may be set by tools like systemd's ``run0`` to display session information.
+
 While you could also just put ``echo`` calls into config.fish, fish_greeting takes care of only being used in interactive shells, so it won't be used e.g. with ``scp`` (which executes a shell), which prevents some errors.
 
 Example
@@ -39,5 +41,5 @@ A simple greeting:
 
   function fish_greeting
       echo Hello friend!
-      echo The time is (set_color yellow)(date +%T)(set_color normal) and this machine is called $hostname
+      echo The time is (set_color yellow)(date +%T)(set_color --reset) and this machine is called $hostname
   end

doc_src/cmds/fish_mode_prompt.rst

@@ -62,7 +62,7 @@ Example
           set_color --bold red
           echo '?'
       end
-      set_color normal
+      set_color --reset
     end