sphinx-build fails with
sphinx.errors.SphinxError: Builder name markdown not registered or available through entry point
Apparently this issue was hidden locally by caching, and not checked
in CI because of this error causing
tests/checks/sphinx-markdown-changelog.fish to be skipped.
sphinx-build 7.2.6
runner@runnervm3ublj:~/work/fish-shell/fish-shell$ python -c 'import sphinx_markdown_builder'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/runner/.local/lib/python3.12/site-packages/sphinx_markdown_builder/__init__.py", line 6, in <module>
from sphinx.util.typing import ExtensionMetadata
ImportError: cannot import name 'ExtensionMetadata' from 'sphinx.util.typing' (/usr/lib/python3/dist-packages/sphinx/util/typing.py)
This reverts commit 7b495497d7.
While at it, fail the test earlier if something went wrong, because the
remaining check will likely also fail and confuse.
GitHub-flavored Markdown translates line breaks to <br/>, which does
not match our intent. Work around that by joining lines when producing
Markdown output.
System tests typically run outside the workspace directory, but they
still have read-only access to the workspace; fix it accordingly.
This test only works on git checkouts, not in tarballs, so skip it
if .git doesn't exist.
The previous version results in an immediate workflow failure due to a
syntax error in the YAML. `workflow_dispatch` should be a dictionary
key, with its value being another dictionary (whose only key is `inputs`
at the moment).
Release automation can be tested on any GitHub fork, using
build_tools/release.sh $version $repository_owner $git_remote
which should work perfectly except for macOS packages (which fail
unless provided GitHub secrets).
People might push tags to their forks, both non-release tags (which
would trigger an early failure in "is-release-tag") or replicas of
our actual release tags (which would create a draft release etc. and
only fail when building macOS packages).
Run on explicit workflow dispatch to make sure it's not triggered by
accident like that.
This means that we'll use the .github/workflows/release.yml from
the default branch (i.e. master), so try to make sure it matches the
version in the release, to prevent accidents.
Closes#11816
These tests require building with the `localize-messages` feature.
If certain translations are updated, this test might fail, either
because a message was changed in the source, or because a translation of
a message was changed, or because a translation was added in a language
which previously did not have a translation for a particular message,
and we rely on that in the test. If any of these happen, the tests need
to be updated accordingly.
Closes#11726
This completely removes our runtime dependency on gettext. As a
replacement, we have our own code for runtime localization in
`src/wutil/gettext.rs`. It considers the relevant locale variables to
decide which message catalogs to take localizations from. The use of
locale variables is mostly the same as in gettext, with the notable
exception that we do not support "default dialects". If `LANGUAGE=ll` is
set and we don't have a `ll` catalog but a `ll_CC` catalog, we will use
the catalog with the country code suffix. If multiple such catalogs
exist, we use an arbitrary one. (At the moment we have at most one
catalog per language, so this is not particularly relevant.)
By using an `EnvStack` to pass variables to gettext at runtime, we now
respect locale variables which are not exported.
For early output, we don't have an `EnvStack` to pass, so we add an
initialization function which constructs an `EnvStack` containing the
relevant locale variables from the corresponding Environment variables.
Treat `LANGUAGE` as path variable. This add automatic colon-splitting.
The sourcing of catalogs is completely reworked. Instead of looking for
MO files at runtime, we create catalogs as Rust maps at build time, by
converting PO files into MO data, which is not stored, but immediately
parsed to extract the mappings. From the mappings, we create Rust source
code as a build artifact, which is then macro-included in the crate's
library, i.e. `crates/gettext-maps/src/lib.rs`. The code in
`src/wutil/gettext.rs` includes the message catalogs from this library,
resulting in the message catalogs being built into the executable.
The `localize-messages` feature can now be used to control whether to
build with gettext support. By default, it is enabled. If `msgfmt` is
not available at build time, and `gettext` is enabled, a warning will be
emitted and fish is built with gettext support, but without any message
catalogs, so localization will not work then.
As a performance optimization, for each language we cache a separate
Rust source file containing its catalog as a map. This allows us to
reuse parsing results if the corresponding PO files have not changed
since we cached the parsing result.
Note that this approach does not eliminate our build-time dependency on
gettext. The process for generating PO files (which uses `msguniq` and
`msgmerge`) is unchanged, and we still need `msgfmt` to translate from
PO to MO. We could parse PO files directly, but these are significantly
more complex to parse, so we use `msgfmt` to do it for us and parse the
resulting MO data.
Advantages of the new approach:
- We have no runtime dependency on gettext anymore.
- The implementation has the same behavior everywhere.
- Our implementation is significantly simpler than GNU gettext.
- We can have localization in cargo-only builds by embedding
localizations into the code.
Previously, localization in such builds could only work reliably as
long as the binary was not moved from the build directory.
- We no longer have to take care of building and installing MO files in
build systems; everything we need for localization to work happens
automatically when building fish.
- Reduced overhead when disabling localization, both in compilation time
and binary size.
Disadvantages of this approach:
- Our own runtime implementation of gettext needs to be maintained.
- The implementation has a more limited feature set (but I don't think
it lacks any features which have been in use by fish).
Part of #11726Closes#11583Closes#11725Closes#11683
The extracted release notes trigger a sphinx warning
/tmp/tmp.V6RGP92nc2/src/index.rst:6:Document may not end with a transition.
which we don't seem to get on the full CHANGELOG.rst.
Let's work around that for now.
That configuration is already tested, but not clippy-checked yet.
This sometimes causes things like unused imports linger on master.
Let's at least enable clippy for stable Rust.
Also do the same build_tools/check.sh; since that script already runs
"cargo test --no-default-features", this shouldn't add much work,
though I didn't check that.
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- No *.asc signature file for the tarball (or for any other release assets)
- No GPG-signed Debian and other OBS packages
To-do:
- remove the corresponding entries from
https://github.com/fish-shell/fish-shell/wiki/Release-checklist
and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Let's hope that this doesn't cause security issues.
Usage:
1. run "build_tools/release.sh $version"; this will create and push
a tag, which kicks off .github/workflows/release.yml
2. wait for the draft release to be created at
https://github.com/fish-shell/fish-shell/releases/tags/$version
3. publish the draft (manually, for now). This should unblock the
last part of the workflow (website updates).
Closes#10449
Incremental usage example:
version=4.0.3
repository_owner=fish-shell
remote=origin
cd ../fish-shell-secondary-worktree
git tag -d $version ||:
git push $remote :$version ||:
git reset --hard origin/Integration_$version
for d in .github build_tools; do {
rm -rf $d
cp -r ../fish-shell/$d .
git add $d
} done
git commit -m 'Backport CI/CD'
echo "See https://github.com/$repository_owner/fish-shell/actions"
echo "See the draft release at https://github.com/$repository_owner/fish-shell/releases/$version"
../fish-shell/build_tools/release.sh $version $repository_owner $remote
We use the MSRV for CI checks, and for deploying to old macOS.
But for static Linux builds , there should be no reason to use an
old Rust version. Let's track stable.
By default, we make every rustup user use our pinned version. This might
not be ideal at this point, for a few reasons:
1. we don't have automatic Rust updates yet (see the parent commit),
so this might unnecessarily install an old version. As a contributor,
this feels irritating (newer versions are usually strictly better).
2. it will use more bandwidth and perhaps other resources during "git-bisect"
scenarios
3. somehow rustup will download things redundantly; it will download "1.89.0"
and "stable" even if they are identical. The user will need to clean
those up at some point, even if they didn't add them explicitly.
See also
https://github.com/fish-shell/fish-shell/pull/11712#issuecomment-3165388330
Part of the motivation for rust-toolchain.toml is probably the regular
(every 6 weeks) failures due to the update check, but that failure has been
removed in the parent commit.
The other motivation ("fix the issue of local compiles running into lint
warnings from newer compilers") is a fair point but I think we should rather
fix warnings quickly.
Let's remove rust-toolchain.toml again until we have more agreement on what
we should do.
This reverts commits
* f806d35af8 (Ignore rust-toolchain.toml in CI, 2025-08-07)
* 9714b98262 (Explicitly use fully qualified rust version numbers, 2025-08-07)
* 921aaa0786 (Add rust-toolchain.toml, 2025-08-07)
Closes#11718
This ensures that, by default, developers use the toolchain that is also tested
in CI, avoiding spurious warnings from lints added in new compiler versions.
Today, when a change introduces warnings, the change author might not see
them. Fix that by making clippy fail on warnings.
AFAICT, "clippy --deny-warnings" will also fail on rustc warnings.
I'd imagine this is what most respectable Rust projects do.
Pin stable rust so we won't get unrelated failures. Alternatively, we could
keep using "dtolnay/rust-toolchain@stable", that should be fine too (as long
as we have the capacity to quickly silence/iron out clippy failures).
While at it, remove some unneeded dependencies. Keep gettext because that
one might enable some cfg-directives (?).
Other cfgs like feature="benchmark" and target_os != "linux" are not yet checked in CI.
See #11584
Extract a github action to reduce the number of references to our MSRV and
stable (to be pinned in the next commit).
While at it, use the MSRV for macOS builds; this means that we'll be less
like accidentally to break the macOS build when bumping the MSRV. I don't
think there is a reason for using 1.73 specifically, other than "it's the
highest we can use on old macOS", so using an even older one should be fine.
Remove dependency on CTest. Parallel execution is handled by `test_driver.py`
internally now, so CTest is no longer relevant for performance.
This also removes CMake targets for single tests. As a replacement,
`test_driver.py` can be called directly with the path to the build directory as
the first argument and the path to the desired test as the second argument.
Ensuring that the executables in the build directory are up to date needs to be
done separately.
For a pure cargo build, an example of running a single test would be:
`cargo b && tests/test_driver.py target/debug tests/checks/abbr.fish`
The recommended way of running tests is `build_tools/check.sh`, which runs more
extensive tests and does not depend on CMake. That script does not work in CI
yet, so CMake testing is retained for now.
Update CI config to use the new `FISH_TEST_MAX_CONCURRENCY`.
Also update the FreeBSD version, since the previous one is outdated and does not
support the semaphore logic in `test_driver.py`.
This new wrapper type can be constructed via macros which invoke the
`gettext_extract` proc macro to extract the string literals for PO file
generation.
The type checking enabled by this wrapper should prevent trying to obtain
translations for a string for which none exist.
Because some strings (e.g. for completions) are not defined in Rust, but rather
in fish scripts, the `LocalizableString` type can also be constructed from
non-literals, in which case no extraction happens.
In such cases, it is the programmer's responsibility to only construct the type
for strings which are available for localization.
This approach is a replacement for the `cargo-expand`-based extraction.
When building with the `FISH_GETTEXT_EXTRACTION_FILE` environment variable set,
the `gettext_extract` proc macro will write the messages marked for extraction
to a file in the directory specified by the variable.
Updates to the po files:
- This is the result of running the `update_translations.fish` script using the
new proc_macro extraction. It finds additional messages compared to the
`cargo-expand` based approach.
- Messages IDs corresponding to paths are removed. The do not have localizations
in any language and localizing paths would not make sense. I have not
investigated how they made it into the po files in the first place.
- Some messages are reordered due to `msguniq` sorting differing from `sort`.
Remove docs about installing `cargo-expand`
These are no longer needed due to the switch to our extraction macro.
clang --version here outputs "clang version 19.1.7"
but it looks like that changed on GHA's Ubuntu runner:
++ clang --version
++ awk 'NR==1 { split($NF, version, "."); print version[1] }'
+ llvm_version='(1ubuntu1)'
which leads to
The CHECK on line 7 wants:
abbr -a -- cuckoo somevalue # imported from a universal variable, see `help abbr`
but there was no remaining output to match.
additional output on stderr:1:111:
=================================================================
==4680==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7bc287a5d402 at pc 0x55e597fa96b6 bp 0x7ffd2bc00700 sp 0x7ffd2bbffea8
READ of size 18 at 0x7bc287a5d402 thread T0
==4680==WARNING: invalid path to external symbolizer!
==4680==WARNING: Failed to use and restart external symbolizer!
Fix that, assuming that "clang --version" always matches the latest symbolizer
that is installed. While at it, leave a "set -x" (which should be the
default for CI), and install llvm explicitly.
Our use of the terminfo database in /usr/share/terminfo/$TERM is both
1. a way for users to configure app behavior in their terminal (by
setting TERM, copying around and modifying terminfo files)
2. a way for terminal emulator developers to advertise support for
backwards-incompatible features that are not otherwise easily observable.
To 1: this is not ideal (it's very easy to break things). There's not many
things that realistically need configuration; let's use shell variables
instead.
To 2: in practice, feature-probing via terminfo is often wrong. There's not
many backwards-incompatible features that need this; for the ones that do
we can still use terminfo capabilities but query the terminal via XTGETTCAP
directly, skipping the file (which may not exist on the same system as
the terminal).
---
Get rid of terminfo. If anyone finds a $TERM where we need different behavior,
we can hardcode that into fish.
* Allow to override this with `fish_features=no-ignore-terminfo fish`
Not sure if we should document this, since it's supposed to be removed soon,
and if someone needs this (which we don't expect), we'd like to know.
* This is supported on a best-effort basis; it doesn't match the previous
behavior exactly. For simplicity of implementation, it will not change
the fact that we now:
* use parm_left_cursor (CSI Ps D) instead of cursor_left (CSI D) if
terminfo claims the former is supported
* no longer support eat_newline_glitch, which seems no longer present
on today's ConEmu and ConHost
* Tested as described in https://github.com/fish-shell/fish-shell/pull/11345#discussion_r2030121580
* add `man fish-terminal-compatibility` to state our assumptions.
This could help terminal emulator developers.
* assume `parm_up_cursor` is supported if the terminal supports XTGETTCAP
* Extract all control sequences to src/terminal_command.rs.
* Remove the "\x1b(B" prefix from EXIT_ATTRIBUTE_MODE. I doubt it's really
needed.
* assume it's generally okay to output 256 colors
Things have improved since commit 3669805627 (Improve compatibility with
0-16 color terminals., 2016-07-21).
Apparently almost every actively developed terminal supports it, including
Terminal.app and GNU screen.
* That is, we default `fish_term256` to true and keep it only as a way to
opt out of the the full 256 palette (e.g. switching to the 16-color
palette).
* `TERM=xterm-16color` has the same opt-out effect.
* `TERM` is generally ignored but add back basic compatiblity by turning
off color for "ansi-m", "linux-m" and "xterm-mono"; these are probably
not set accidentally.
* Since `TERM` is (mostly) ignored, we don't need the magic "xterm" in
tests. Unset it instead.
* Note that our pexpect tests used a dumb terminal because:
1. it makes fish do a full redraw of the commandline everytime, making it
easier to write assertions.
2. it disables all control sequences for colors, etc, which we usually
don't want to test explicitly.
I don't think TERM=dumb has any other use, so it would be better
to print escape sequences unconditionally, and strip them in
the test driver (leaving this for later, since it's a bit more involved).
Closes#11344Closes#11345
We can no longer have a single template that is always inserted. Instead
the web UI shows a menu allowing to select between one of our templates and
a blank one.
