This completely removes our runtime dependency on gettext. As a
replacement, we have our own code for runtime localization in
`src/wutil/gettext.rs`. It considers the relevant locale variables to
decide which message catalogs to take localizations from. The use of
locale variables is mostly the same as in gettext, with the notable
exception that we do not support "default dialects". If `LANGUAGE=ll` is
set and we don't have a `ll` catalog but a `ll_CC` catalog, we will use
the catalog with the country code suffix. If multiple such catalogs
exist, we use an arbitrary one. (At the moment we have at most one
catalog per language, so this is not particularly relevant.)
By using an `EnvStack` to pass variables to gettext at runtime, we now
respect locale variables which are not exported.
For early output, we don't have an `EnvStack` to pass, so we add an
initialization function which constructs an `EnvStack` containing the
relevant locale variables from the corresponding Environment variables.
Treat `LANGUAGE` as path variable. This add automatic colon-splitting.
The sourcing of catalogs is completely reworked. Instead of looking for
MO files at runtime, we create catalogs as Rust maps at build time, by
converting PO files into MO data, which is not stored, but immediately
parsed to extract the mappings. From the mappings, we create Rust source
code as a build artifact, which is then macro-included in the crate's
library, i.e. `crates/gettext-maps/src/lib.rs`. The code in
`src/wutil/gettext.rs` includes the message catalogs from this library,
resulting in the message catalogs being built into the executable.
The `localize-messages` feature can now be used to control whether to
build with gettext support. By default, it is enabled. If `msgfmt` is
not available at build time, and `gettext` is enabled, a warning will be
emitted and fish is built with gettext support, but without any message
catalogs, so localization will not work then.
As a performance optimization, for each language we cache a separate
Rust source file containing its catalog as a map. This allows us to
reuse parsing results if the corresponding PO files have not changed
since we cached the parsing result.
Note that this approach does not eliminate our build-time dependency on
gettext. The process for generating PO files (which uses `msguniq` and
`msgmerge`) is unchanged, and we still need `msgfmt` to translate from
PO to MO. We could parse PO files directly, but these are significantly
more complex to parse, so we use `msgfmt` to do it for us and parse the
resulting MO data.
Advantages of the new approach:
- We have no runtime dependency on gettext anymore.
- The implementation has the same behavior everywhere.
- Our implementation is significantly simpler than GNU gettext.
- We can have localization in cargo-only builds by embedding
localizations into the code.
Previously, localization in such builds could only work reliably as
long as the binary was not moved from the build directory.
- We no longer have to take care of building and installing MO files in
build systems; everything we need for localization to work happens
automatically when building fish.
- Reduced overhead when disabling localization, both in compilation time
and binary size.
Disadvantages of this approach:
- Our own runtime implementation of gettext needs to be maintained.
- The implementation has a more limited feature set (but I don't think
it lacks any features which have been in use by fish).
Part of #11726Closes#11583Closes#11725Closes#11683
Things that are not currently happening in this workflow:
- No GPG-signature on the Git tag
- No *.asc signature file for the tarball (or for any other release assets)
- No GPG-signed Debian and other OBS packages
To-do:
- remove the corresponding entries from
https://github.com/fish-shell/fish-shell/wiki/Release-checklist
and link to this workflow.
- Maybe add some testing (for the Linux packages)?.
- Let's hope that this doesn't cause security issues.
Usage:
1. run "build_tools/release.sh $version"; this will create and push
a tag, which kicks off .github/workflows/release.yml
2. wait for the draft release to be created at
https://github.com/fish-shell/fish-shell/releases/tags/$version
3. publish the draft (manually, for now). This should unblock the
last part of the workflow (website updates).
Closes#10449
Incremental usage example:
version=4.0.3
repository_owner=fish-shell
remote=origin
cd ../fish-shell-secondary-worktree
git tag -d $version ||:
git push $remote :$version ||:
git reset --hard origin/Integration_$version
for d in .github build_tools; do {
rm -rf $d
cp -r ../fish-shell/$d .
git add $d
} done
git commit -m 'Backport CI/CD'
echo "See https://github.com/$repository_owner/fish-shell/actions"
echo "See the draft release at https://github.com/$repository_owner/fish-shell/releases/$version"
../fish-shell/build_tools/release.sh $version $repository_owner $remote
Not sure about whether "man fish-terminal-compatibility"; it's not
really meant for end-users, but it also doesn't hurt raise awareness
of the existence of this doc.
Either way, we should be consistent with embedded builds, where this
works since the parent commit.
Remove dependency on CTest. Parallel execution is handled by `test_driver.py`
internally now, so CTest is no longer relevant for performance.
This also removes CMake targets for single tests. As a replacement,
`test_driver.py` can be called directly with the path to the build directory as
the first argument and the path to the desired test as the second argument.
Ensuring that the executables in the build directory are up to date needs to be
done separately.
For a pure cargo build, an example of running a single test would be:
`cargo b && tests/test_driver.py target/debug tests/checks/abbr.fish`
The recommended way of running tests is `build_tools/check.sh`, which runs more
extensive tests and does not depend on CMake. That script does not work in CI
yet, so CMake testing is retained for now.
Update CI config to use the new `FISH_TEST_MAX_CONCURRENCY`.
Also update the FreeBSD version, since the previous one is outdated and does not
support the semaphore logic in `test_driver.py`.
This was only used to cache `fish_test_driver`, which can be built in a few tens
of milliseconds on most systems.
This avoids using outdated binaries for testing and simplifies the code and its
usage.
FindRust is too clever by half. It tries to do rustup's job for it.
See b38551dde9 for how that can break.
So we simplify it, and only let it check three things:
- Where's rustc? Look in $PATH and ~/.cargo/bin
- Where's cargo? Look in $PATH and ~/.cargo/bin
- What is the rust target (because we pass it explicitly)?
If any of these aren't that simple, we'll ask the user to tell us,
by setting Rust_COMPILER, Rust_CARGO or Rust_CARGO_TARGET.
None of the other things are helpful to us - we do not support windows
or whatever a "unikraft" is, and if the rust version doesn't work
it'll print its own error.
We could add a rustc version check, but that will become less and less
useful because rustc versions since 1.56 (released October 2021) will check rust-version in
Cargo.toml. So even at this point it's only pretty old rust versions already.
rustup has changed its output for 'rustup toolchain list --verbose`.
Teach FindRust.cmake about it, so that it may shamble on.
This was reported against BSD and also affects macOS and ALSO affects Linux; our
CI just doesn't have a new enough rustup. Anyways we can't have nice things.
Saturating all cores easily leads to timeouts.
The system might be doing something else, and it only leaves at most
one core per-test.
E.g. this will cause 90% of runs to fail on a raspberry pi 4.
Note that we set CTEST_PARALLEL_LEVEL on our CI machines anyway, so
this will not affect them.
This replaces the test_driver.sh/test.fish/interactive.fish system with a test driver written in python that calls into littlecheck directly and runs pexpect in a subprocess.
This means we reduce the reliance on the fish that we're testing, and we remove a posix sh script that is a weird stumbling block (see my recent quest to make it work on directories with spaces).
To run specific tests, e.g. all the tmux tests and bind.py:
tests/test_driver.py target/release/ tests/checks/tmux*.fish tests/pexpects/bind.py
I think the dynamic detection patch ends up overriding the environment variable
set by CI (if present), because `if(NOT CTEST_PARALLEL_LEVEL)` would define to
false even if an environment variable of that name existed then we end up
explicitly assigning the environment variable by that name upon invocation with
`env`.
The default is still "../test/root/bin/", but we now pass this
through,
so you *can* run
`FISHDIR=$PWD ../tests/test_driver.sh $PWD/../tests/test.fish`
Unfortunately it does not appear like #[cfg(test)] works for build.rs?
Investigating a better solution, but this is a good idea anyway (or `make
test` would generate man pages via build.rs!)
We turned it off, but for some reason (cmake version?) that stopped working on my system.
So instead we just remove all the code that does it.
To be honest I do not know why this exists anyway.
CMake Warning (dev) at cmake/Tests.cmake:56 (add_custom_command):
Exactly one of PRE_BUILD, PRE_LINK, or POST_BUILD must be given. Assuming
POST_BUILD to preserve backward compatibility.
Policy CMP0175 is not set: add_custom_command() rejects invalid arguments.
Run "cmake --help-policy CMP0175" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.
So we just keep it the same.
Commit 4e79ec5f tried to restore the static PCRE2 build after the update to the
pcre2 crate, but it set an environment variable at configure time, not build
time.
Properly set the environment variable at build time.
The recent update to the rust-pcre2 crate lost the property where a static
PCRE2 build could be enabled with a Cargo feature. This means that static
PCRE2 builds can no longer be forced.
Switch to setting the "PCRE2_SYS_STATIC" variable again, which is how the
official rust-pcre2 crate expects to work.
fish adds ~/.local/share/fish/man to its MANPATH for builtins etc. But pages
like fish-doc are unambiguous so it seems like they should be accessible
from outside fish by default.
Closes#10711
- Ubuntu focal is the lowest LTS release that we can support with only
distro packages (e.g. no rustup).
- Remove tsan from Cirrus (it's not working currently, and also not really
important).
- Remove Centos (it passes tests but I'm not sure it's worth adding; there
isn't even an official docker image for CentOS Stream).
The CMake `cargo test` integration was broken if Rust_CARGO_TARGET were used
with `CARGO_FLAGS` set to `-Zbuild-std` (e.g. to target i586 under i686 without
the i586 toolchain installed).
See the changelog additions for user-visible changes.
Since we enable/disable terminal protocols whenever we pass terminal ownership,
tests can no longer run in parallel on the same terminal.
For the same reason, readline shortcuts in the gdb REPL will not work anymore.
As a remedy, use gdbserver, or lobby for CSI u support in libreadline.
Add sleep to some tests, otherwise they fall (both in CI and locally).
There are two weird failures on FreeBSD remaining, disable them for now
https://github.com/fish-shell/fish-shell/pull/10359/checks?check_run_id=23330096362
Design and implementation borrows heavily from Kakoune.
In future, we should try to implement more of the kitty progressive
enhancements.
Closes#10359
We are no longer C++, we no longer support xcode
Note: This will remove a warning "DO NOT EDIT" comment from __fish_build_paths.fish, but
that's unnecessary. The file is typically in /usr or another
package-manager-owned location, so people don't typically edit it.
And if it did we don't actually *care*, it'll work fine.
