Another reduction in size of the main crate. Also allows other crates to
depend on the new wchar crate.
The original `src/wchar.rs` file is kept around for now to keep the
prelude imports working.
Part of #12182
Dependencies between crates must form a DAG. This means that breaking up
the large library crate requires breaking dependency cycles. The goal of
this commit is creating a crate which contains some of the main crate's
functionality, without depending on the main crate.
To start off, we only move things required for extracting `src/wchar.rs`
and `src/wchar_ext.rs`, which will happen in a subsequent commit.
Part of #12182
This should help with improving incremental build speed. Extracting this
code is easy, since it does not have dependencies. It also unblocks
further extraction of code which depends on widecharwidth.
Closes#12181
This is part of the larger effort of splitting up fish's huge main crate
to improve incremental build speed.
We could extract more logic from `src/wutil/gettext.rs` into the new
crate, but this would require putting wide-string handling into that
crate, which I'm not sure we want. Doing so would have the advantage
that crates which don't depend on fish's main crate (i.e. all crates
other than fish's main crate itself and the binary crates built on top
of it) could then localize messages as well. This will be less relevant
if we replace gettext with Fluent for messages originating from the Rust
sources.
Closes#12108
Implicitly-universal variables have some downsides:
- It's surprising that "set fish_color_normal ..."
and "set fish_key_bindings fish_vi_key_bindings" propagate to other
shells and persist, especially since all other variables (and other
shells) would use the global scope.
- they don't play well with tracking configuration in Git.
- we don't know how to roll out updates to the default theme (which is
problematic since can look bad depending on terminal background
color scheme).
It's sort of possible to use only globals and unset universal variables
(because fish only sets them at first startup), but that requires
knowledge of fish internals; I don't think many people do that.
So:
- Set all color variables that are not already set as globals.
- To enable this do the following, once, after upgrading:
copy any existing universal color variables to globals, and:
- if existing universal color variables exactly match
the previous default theme, and pretend they didn't exist.
- else migrate the universals to ~/.config/fish/conf.d/fish_frozen_theme.fish,
which is a less surprising way of persisting this.
- either way, delete all universals to do the right thing for most users.
- Make sure that webconfig's "Set Theme" continues to:
- instantly update all running shells
- This is achieved by a new universal variable (but only for
notifying shells, so this doesn't actually need to be persisted).
In future, we could use any other IPC mechanism such as "kill -SIGUSR1"
or if we go for a new feature, "varsave" or "set --broadcast", see
https://github.com/fish-shell/fish-shell/issues/7317#issuecomment-701165897https://github.com/fish-shell/fish-shell/pull/8455#discussion_r757837137.
- persist the theme updates, completely overriding any previous theme.
Use the same "fish_frozen_theme.fish" snippet as for migration (see above).
It's not meant to be edited directly. If people want flexibility
the should delete it.
It could be a universal variable instead of a conf snippet file;
but I figured that the separate file looks nicer
(we can have better comments etc.)
- Ask the terminal whether it's using dark or light mode, and use an
optimized default. Add dark/light variants to themes,
and the "unknown" variant for the default theme.
Other themes don't need the "unknown" variant;
webconfig already has a background color in context,
and CLI can require the user to specify variant explicitly if
terminal doesn't advertise colors.
- Every variable that is set as part of fish's default behavior
gets a "--label=default" tacked onto it.
This is to allow our fish_terminal_color_theme event handler to
know which variables it is allowed to update. It's also necessary
until we revert 7e3fac561d (Query terminal only just before reading
from it, 2025-09-25) because since commit, we need to wait until
the first reader_push() to get query results. By this time, the
user's config.fish may already have set variables.
If the user sets variables via either webconfig, "fish_config theme
{choose,save}", or directly via "set fish_color_...", they'd almost
always remove this label.
- For consistency, make default fish_key_bindings global
(note that, for better or worse, fish_add_path still remains as
one place that implicitly sets universal variables, but it's not
something we inject by default)
- Have "fish_config theme choose" and webconfig equivalents reset
all color variables. This makes much more sense than keeping a
hardcoded subset of "known colors"; and now that we don't really
expect to be deleting universals this way, it's actually possible
to make this change without much fear.
Should have split this into two commits (the changelog entries are
intertwined though).
Closes#11580Closes#11435Closes#7317
Ref: https://github.com/fish-shell/fish-shell/issues/12096#issuecomment-3632065704
Multiple gettext-extraction proc macro instances can run at the same
time due to Rust's compilation model. In the previous implementation,
where every instance appended to the same file, this has resulted in
corruption of the file. This was reported and discussed in
https://github.com/fish-shell/fish-shell/pull/11928#discussion_r2488047964
for the equivalent macro for Fluent message ID extraction. The
underlying problem is the same.
The best way we have found to avoid such race condition is to write each
entry to a new file, and concatenate them together before using them.
It's not a beautiful approach, but it should be fairly robust and
portable.
Closes#12125
This eliminates the proc-macro panic occasionally observed in CI,
instead ignoring paths which cannot be canonicalized.
See #12120.
While this does not address the underlying issue of why the proc-macro
fail happens, it should result in builds no longer failing, and since
they failed in the case where no embedding is desired, there should be
no issue with ignoring failed path canonicalization, since we do not
want to embed anything in these cases.
The `mkstemp` function opens files without setting `O_CLOEXEC`. We could
manually set this using `fnctl` once the file is opened, but that has
the issue of introducing race conditions. If fish `exec`s in another
thread before the `fnctl` call completes, the file would be left open.
One way of mitigating this is `mkostemp`, but that function is not
available on all systems fish supports, so we can't rely on it.
Instead, build our own tempfile creation logic which uses the `rand`
crate for getting entropy and relies on Rust's stdlib for the rest.
The stdlib functions we use set `O_CLOEXEC` by default.
For directory creation we keep using `mkdtemp`, since there we don't
open anything. We could replace this by extending our custom logic a
bit, which would allow us to drop the `nix` dependency for our
`tempfile` crate, but since the code is simpler as it is now and we need
nix in fish's main crate, there is no need to modify the directory
creation code.
Part of #12030
`SmallRng` was chosen in part due to limitation of old macOS versions.
This is no longer relevant, since the affected versions are not
supported by Rust anymore.
Switch everything which does not need fixed sequences based on a seed to
`ThreadRng`, which has better cryptographic properties and is
occasionally reseeded. Performance differences should not matter much,
since initialization of `ThreadRng` is not that expensive and it's
generation speed is easily fast enough for our purposes.
In some cases, like tests and the `random` builtin, we want a PRNG which
consistently produces the same sequence of values for a given seed.
`ThreadRng` does not do this, since it is occasionally reseeded
automatically. In these cases, we keep using `SmallRng`.
Part of #12030
Function names containing `gen` have been deprecated to avoid conflicts
with the Rust 2024 keyword, so this commit also switches to the new
names.
Part of #12030
While it's not necessary to rebuild the proc macro for extraction when
the env var controlling the output location changes, this way everything
using the macro will automatically be rebuilt when this env var changes,
making it impossible to forget adding this to other `build.rs` files.
For now, keeping the rebuild instructions in fish's main crate's
`build.rs` would be fine as well, but if we start breaking this crate
into smaller parts, it would become annoying to add the rebuild command
in every crate depending on gettext extraction.
Closes#12107
ja: the motivation for our own crate is
1. the tempfile crate is probably overkill for such a small
piece of functionality (given that we already assume Unix)
2. we want to have full control over the few temp files we
do create
Closes#12028
Today, this file is only supported in CMake builds. This is the only
place where CMake builds currently need $__fish_data_dir as opposed
to using "status get-file".
Let's embed __fish_build_paths so we can treat it like other assets.
This enables users of "embed-data" to do "rm -rf $__fish_data_dir"
(though that might break plugins).
This completely removes our runtime dependency on gettext. As a
replacement, we have our own code for runtime localization in
`src/wutil/gettext.rs`. It considers the relevant locale variables to
decide which message catalogs to take localizations from. The use of
locale variables is mostly the same as in gettext, with the notable
exception that we do not support "default dialects". If `LANGUAGE=ll` is
set and we don't have a `ll` catalog but a `ll_CC` catalog, we will use
the catalog with the country code suffix. If multiple such catalogs
exist, we use an arbitrary one. (At the moment we have at most one
catalog per language, so this is not particularly relevant.)
By using an `EnvStack` to pass variables to gettext at runtime, we now
respect locale variables which are not exported.
For early output, we don't have an `EnvStack` to pass, so we add an
initialization function which constructs an `EnvStack` containing the
relevant locale variables from the corresponding Environment variables.
Treat `LANGUAGE` as path variable. This add automatic colon-splitting.
The sourcing of catalogs is completely reworked. Instead of looking for
MO files at runtime, we create catalogs as Rust maps at build time, by
converting PO files into MO data, which is not stored, but immediately
parsed to extract the mappings. From the mappings, we create Rust source
code as a build artifact, which is then macro-included in the crate's
library, i.e. `crates/gettext-maps/src/lib.rs`. The code in
`src/wutil/gettext.rs` includes the message catalogs from this library,
resulting in the message catalogs being built into the executable.
The `localize-messages` feature can now be used to control whether to
build with gettext support. By default, it is enabled. If `msgfmt` is
not available at build time, and `gettext` is enabled, a warning will be
emitted and fish is built with gettext support, but without any message
catalogs, so localization will not work then.
As a performance optimization, for each language we cache a separate
Rust source file containing its catalog as a map. This allows us to
reuse parsing results if the corresponding PO files have not changed
since we cached the parsing result.
Note that this approach does not eliminate our build-time dependency on
gettext. The process for generating PO files (which uses `msguniq` and
`msgmerge`) is unchanged, and we still need `msgfmt` to translate from
PO to MO. We could parse PO files directly, but these are significantly
more complex to parse, so we use `msgfmt` to do it for us and parse the
resulting MO data.
Advantages of the new approach:
- We have no runtime dependency on gettext anymore.
- The implementation has the same behavior everywhere.
- Our implementation is significantly simpler than GNU gettext.
- We can have localization in cargo-only builds by embedding
localizations into the code.
Previously, localization in such builds could only work reliably as
long as the binary was not moved from the build directory.
- We no longer have to take care of building and installing MO files in
build systems; everything we need for localization to work happens
automatically when building fish.
- Reduced overhead when disabling localization, both in compilation time
and binary size.
Disadvantages of this approach:
- Our own runtime implementation of gettext needs to be maintained.
- The implementation has a more limited feature set (but I don't think
it lacks any features which have been in use by fish).
Part of #11726Closes#11583Closes#11725Closes#11683
Building man pages takes significant time due to Sphinx running for several
seconds, even when no updates are required. Previously, we added custom logic to
avoid calling `sphinx-build` if the inputs to `sphinx-build` had not changed
since a cached timestamp.
By moving this into its own crate, we can tell cargo to rebuild when the input
files changed and unrelated changes will have no effect on this crate. This
allows us to get rid of the custom code for tracking whether to recompile, while
keeping the effect of only calling `sphinx-build` when appropriate.
In order to avoid code duplication, a new `build-helper` crate is added,
which contains some functionality for use in `build.rs`.
Closes#11737
This new wrapper type can be constructed via macros which invoke the
`gettext_extract` proc macro to extract the string literals for PO file
generation.
The type checking enabled by this wrapper should prevent trying to obtain
translations for a string for which none exist.
Because some strings (e.g. for completions) are not defined in Rust, but rather
in fish scripts, the `LocalizableString` type can also be constructed from
non-literals, in which case no extraction happens.
In such cases, it is the programmer's responsibility to only construct the type
for strings which are available for localization.
This approach is a replacement for the `cargo-expand`-based extraction.
When building with the `FISH_GETTEXT_EXTRACTION_FILE` environment variable set,
the `gettext_extract` proc macro will write the messages marked for extraction
to a file in the directory specified by the variable.
Updates to the po files:
- This is the result of running the `update_translations.fish` script using the
new proc_macro extraction. It finds additional messages compared to the
`cargo-expand` based approach.
- Messages IDs corresponding to paths are removed. The do not have localizations
in any language and localizing paths would not make sense. I have not
investigated how they made it into the po files in the first place.
- Some messages are reordered due to `msguniq` sorting differing from `sort`.
Remove docs about installing `cargo-expand`
These are no longer needed due to the switch to our extraction macro.
After nix updated to 0.30, all functions related to file descriptor accepts impl AsFd, e.g., BorrowedFd. This PR is a minimal update. It tries to use impl AsFd as long as possible, but uses BorrowedFd in some places. Yes it introduces unsafe, but doesn't introduce new unsafe code.
Specifically, the width and precision format specifiers are interpreted as
referring to the width of the grapheme clusters rather than the byte count of
the string. Note that grapheme clusters can differ in width.
If a precision is specified for a string, meaning its "maximum number of
characters", we consider this to limit the width displayed.
If there is a grapheme cluster whose width is greater than 1,
it might not be possible to get precisely the desired width.
In such cases, this last grapheme cluster is excluded from the output.
Note that the definitions used here are not consistent with the `string length`
builtin at the moment, but this has already been the case.
When built with the default "installable" feature, the data files (share/) are
included in the fish binary itself.
Run `fish --install` or `fish --install=noconfirm` (for
non-interactive use) to install fish's data files into ~/.local/share/fish/install
To figure out if the data files are out of date, we write the current version
to a file on install, and read it on start.
CMake disables the default features so nothing changes for that, but this allows installing via `cargo install`,
and even making a static binary that you can then just upload and have extract itself.
We set $__fish_help_dir to empty for installable builds, because we do not have
a way to generate html docs (because we need fish_indent for highlighting).
The man pages are found via $__fish_data_dir/man
