There is no reason to have this file clutter the repo root.
Move it into the `po` directory, and give it a more descriptive name.
Inspired by this discussion:
https://github.com/fish-shell/fish-shell/pull/11463#discussion_r2083453275
I use `template.po` instead of `message.po-template` to be more compatible with
automatic filetype detection. (e.g. vim/nvim detect the former as a po file, but
the latter as a conf file)
This allows msgfmt to detect issues with translations of format strings.
The detection used here is very simple. It just checks if a string contains '%',
and if it does, the entry in the po file is preceded by '#, c-format'.
Any entries with this marker are checked by msgfmt in our tests, so if an issue
arises, we will notice before it is merged.
The old version just prints the entire command being profiled as-is.
If such a command consists of more than one line, these lines do not have any
padding, and thus visually interfere with the timings.
This commit adds padding, such that all lines but the first one have padding
prepended, such that the original line content starts at the column in which the
first line starts.
This does not work perfectly for subcommands (in the profiling sense,
where the command starts with (regex) '-+>' instead of just '>').
In such cases, even if the command string is indented in the source, the command
will not start with whitespace. However, subsequent lines are not trimmed, so
the might be indented farther than they should be relative to the first line of
the command.
Another regression from d51f669647 (Vi mode: avoid placing cursor beyond last
character, 2024-02-14) "Unfortunately Vi mode sometimes needs to temporarily
select past end". So do the replace_one mode bindings which were forgotten.
Fix this.
This surfaces a tricky problem: when we use something like
bind '' self-insert some-command
When key event "x" matches this generic binding, we insert both "self-insert"
and "some-command" at the front of the queue, and do *not* consume "x",
since the binding is empty.
Since there is a command (that might call "exit"), we insert a check-exit
event too, after "self-insert some-command" but _before_ "x".
The check-exit event makes "self-insert" do nothing. I don't think there's a
good reason for this; self-insert can only be triggered by a key event that
maps to self-insert; so there must always be a real key available for it to
consume. A "commandline -f self-insert" is a nop. Skip check-exit here.
Fixes#11484
The purpose of this script is to simplify the translation-related workflow for
both developers and translators. It runs the xgettext, msgmerge, msgfmt pipeline
(or only parts of it, depending on the arguments), either for all languages, or
for one specific one.
Developers can use the script with the `--no-mo` flag to update the PO files for
all languages after changes to the Rust/fish sources, to keep the translations
up to date. Ideally, this would run automatically for all changes, such that
translations are always up to date, but for now, it would already be an
improvement to run this script before releasing a new version of fish.
Translators can use the script in the same way as developers, to get up to date
PO files. To see their translations in action, the script can be called with
`--only-mo`, which takes the current version of the PO files and generates MO
files from them, which get placed in a location which fish (built with `cargo
build`) can detect.
Translators might also find it useful to specify the language they want to work
on as a non-option argument. This argument should be the path to the po file
they want to work on. Specifying non-existing files to work on a new language is
allowed, but the files must be in the po directory and follow the naming
convention.
This simplifies the logic a bit and performs a better.
Performance improvements for extract_fish_script_messages (time in
microseconds):
- explicit regex: from 128241 to 83471 (speedup 1.5)
- implicit regex: from 682203 to 463635 (speedup 1.5)
The replaces the `strs` list by a corresponding file, which eliminates the need
for looping over the list.
Use sed to transform strings into gettext po format entries.
Format the file with fish_indent and use more expressive variable name for the
file cargo expand writes to.
Performance improvements (in microseconds):
- sort+format rust strings: from 21750 to 11096 (speedup 2.0)
The fish builtin string functions are significantly slower than grep + sed.
The final replacement of \' to ' also does not make any sense here, because
single quotes appear unescaped in Rust strings.
Performance improvement: from 404880 to 44843 (speedup 9.0)
Profiling details (from separate runs):
Time (μs) Sum (μs) Command
174 404880 > set -a strs (string match -rv 'BUILD_VERSION:|PACKAGE_NAME' <$tmpfile |
string match -rg 'const [A-Z_]*: &str = "(.*)"' | string replace -a "\'" "'")
404706 404706 -> string match -rv 'BUILD_VERSION:|PACKAGE_NAME' <$tmpfile |
string match -rg 'const [A-Z_]*: &str = "(.*)"' | string replace -a "\'" "'"
202 44843 > set -a strs (grep -Ev 'BUILD_VERSION:|PACKAGE_NAME' <$tmpfile |
grep -E 'const [A-Z_]*: &str = "(.*)"' |
sed -E -e 's/^.*const [A-Z_]*: &str = "(.*)".*$/\1/' -e "s_\\\'_'_g")
4952 44641 -> grep -Ev 'BUILD_VERSION:|PACKAGE_NAME' <$tmpfile |
grep -E 'const [A-Z_]*: &str = "(.*)"' |
sed -E -e 's/^.*const [A-Z_]*: &str = "(.*)".*$/\1/' -e "s_\\\'_'_g"
28716 28716 --> command grep --color=auto $argv
10973 10973 --> command grep --color=auto $argv
Using a file is significantly faster.
Profiling overview (times in microseconds):
- cargo expand: from 4959320 to 4503409 (speedup 1.1)
- gettext call pipeline: from 436996 to 13536 (speedup 32.3)
- static string pipeline: from 477429 to 404880 (speedup 1.18)
My phone uses dotted underline to indicate errors; that seems nice, a bit
less aggressive than curly. Unfortunately dotted underlines are not as well
supported in terminal emulators; sometimes they are barely visible. So it's
unlikely that we want to use --underline=dotted for an important theme.
Add double and dashed too I guess, even though I don't have a concrete
use case..
Commit f4503af037 (Make alt-{b,f} move in directory history if commandline is
empty, 2025-01-06) had the intentional side effect of making alt-{left,right}
(move in directory history) work in Terminal.app and Ghostty without other,
less reliable workarounds.
That commit says "that [workaround] alone should not be the reason for
this change."; maybe this was wrong.
Extend the workaround to Vi mode. The intention here is to provide
alt-{left,right} in Vi mode. This also adds alt-{b,f} which is odd but
mostly harmless (?) because those don't do anything else in Vi mode.
It might be confusing when studying "bind" output but that one already has
almost 400 lines for Vi mode.
Closes#11479
For backwards compatibility, fish does not treat "{echo,hello}" as a compound
statement but as brace expansion (effectively "echo hello"). We interpret
"{X...}" as compound statement only if X is whitespace or ';' (which is an
interesting solution).
A brace expansion at the very start of a command
is usually pointless (space separation is shorter).
The exception are cases where the command name and the first few arguments
share a suffix.
$ {,1,2,3,4}echo
1echo 2echo 3echo 4echo
Not sure if anyone uses anything like that. Perhaps we want to trade
compatibility for simplicity. I don't have a strong opinion on this.
Always parse the opening brace as first character of a command token as
compound statement.
Brace expansion can still be used with a trick like: «''{echo,foo}»
Closes#11477
Source locations (file name and line number) where a string originates is not
required by gettext tooling. It can help translators to identify context,
but the value of this is reduced by our lack of context support, meaning that
all occurrences of a string will receive the same translation.
Translators can use `rg` or similar tools to find the source locations.
For further details see this thread:
https://github.com/fish-shell/fish-shell/pull/11463#discussion_r2079378627
The main advantage is that updates to the PO files are now only necessary when
the source strings change, which greatly reduces the diff noise.
A secondary benefit is that the string extraction logic is simplified.
We can now directly extract the strings from fish scripts,
and several issues are fixed alongside, mostly related to quoting.
The regex for extracting implicit messages from fish scripts has been tweaked to
ignore commented-out lines, and properly support lines starting with `and`/`or`.
The old regex has the problem that it does not handle lines containing any
non-space characters in front of ` complete` (or ` function`), which results in
`string replace` leaving this part in the resulting string.
For example,
`and complete -d "foo"`
would turn into
`andN_ foo`
if passed to
`string replace --regex $regex 'N_ $1'` (where `$regex` is the `$implicit_regex`) variable.
Another issue are commented-out lines.
This was added without a use case. Now there is a use case (#11084) that
needs it to include all tokens which the sister option "--tokens-expanded"
should not do. The inconsistency is probably not worth it, given that this
"--tokens-raw" use case can be served by a future "read -zal --tokenize-raw".
completions frequently use
argparse ... -- (commandline -xpc)
The "commandline -xpc" output
contains only string tokens.
A syntactically-valid process ("-p") consistes of only string tokens
and redirection tokens. We skip all non-string tokens, but we do include
redirection targets, which are always strings. This is weird, and confuses
completion scripts such as the one above. Leave out redirection targets too.
Part of #11084
Squashed commit of the following:
commit 23163d40bed2d97c72050990cf15db3944ce2ff0
Author: DaiLu Cao <caodailu@foxmail.com>
Date: Thu Apr 24 10:11:04 2025 +0800
Manually review up to line 1055 and modify all Chinese colons '：' to English ':' to prevent potential unknown errors.
commit dca5fb8182b94bffab5034dc5626b2b98d026b6f
Author: DaiLu Cao <caodailu@foxmail.com>
Date: Thu Apr 17 10:50:13 2025 +0800
Manually proofread up to 340 lines
commit 4b2d91c1138f3c8dec15b68aeb0510f02e15a776
Author: DaiLu Cao <caodailu@foxmail.com>
Date: Thu Apr 17 09:50:21 2025 +0800
use msgfmt check and fix all error
commit e2470d81c01ab7bf46d3d6ffd0291a05d4b38e13
Author: DaiLu Cao <caodailu@foxmail.com>
Date: Wed Apr 16 11:10:54 2025 +0800
Fix translation error converting '\\n' to '\\ n' error
commit 7ff970d06ce950aee35e1fb0ec70338f7bd42c1d
Author: DaiLu Cao <caodailu@foxmail.com>
Date: Wed Apr 16 10:53:38 2025 +0800
Fix make error, local cargo test completed
commit 018dfa225530a85486903ef58d47f4c358956b0b
Author: DaiLu Cao <caodailu@foxmail.com>
Date: Fri Apr 11 16:46:36 2025 +0800
modification of make errors
commit cbebd506a500aecc0669dce7f08422fcfed5615f
Author: DaiLu Cao <caodailu@foxmail.com>
Date: Fri Apr 11 15:45:01 2025 +0800
The second modification of make errors are all symbol problems
commit f75c3f7a2a84ffaea4eb642532b5a24da1c9154f
Author: DaiLu Cao <caodailu@foxmail.com>
Date: Fri Apr 11 15:27:01 2025 +0800
Re-add Chinese translation, try to solve the problem of make failing
commit 58551be20d261e3466a9e4ede290675f633e94a3
Author: DaiLu Cao <caodailu@foxmail.com>
Date: Fri Apr 11 15:06:01 2025 +0800
Supplement Chinese translation
This greatly reduces the number of changes necessary to the PO files when the
Rust/fish source files are updated. (Changes to the line number can be applied
automatically, but this adds a lot of noise to the git history.)
Due to the way we have been extracting Rust strings, differentiation between
the same source string in different contexts has not been possible regardless
of the change.
It seems that duplicate msgid entries are not permitted in PO files, so since we
do not use context to distinguish the strings we extract, there is no way to
have context-/location-dependent translations, so we might as well reduce the
git noise by eliminating line numbers.
Including source locations helps translators with understanding context.
Because we do not distinguish between contexts for a given source string,
this is of limited utility, but keeping file names at least allows to open the
relevant files and search them for the string. This might also be helpful to
identify translations which do not make sense in all context in which they are
used. (Although without adding context support, the only remedy would be to
remove the translation altogether, as far as I can tell.)
For extraction from Rust, additional issues are fixed:
- File name extraction from the grep results now works properly. Previously,
lines not starting with whitespace resulted in missing or corrupted matches.
(missing if the source line contains no colon followed by a whitespace,
corrupted if it does, then the match included the part of the line in front of
the colon, instead of just the location)
- Only a single source location per string was supported (`head -n1`). The new
approach using sed does not have this limitation.
Commit daa692a20b (Remove unnecessary escaping for # and ~ inside key name
tokens, 2025-04-01) stopped escaping ? in fish_key_reader output. This is
generally correct but not if the "qmark-noglob" feature flag is turned off.
Add that back, to be safe.
While at it, pass an environment variable more explicitly in a test.
