Our use of the terminfo database in /usr/share/terminfo/$TERM is both
1. a way for users to configure app behavior in their terminal (by
setting TERM, copying around and modifying terminfo files)
2. a way for terminal emulator developers to advertise support for
backwards-incompatible features that are not otherwise easily observable.
To 1: this is not ideal (it's very easy to break things). There's not many
things that realistically need configuration; let's use shell variables
instead.
To 2: in practice, feature-probing via terminfo is often wrong. There's not
many backwards-incompatible features that need this; for the ones that do
we can still use terminfo capabilities but query the terminal via XTGETTCAP
directly, skipping the file (which may not exist on the same system as
the terminal).
---
Get rid of terminfo. If anyone finds a $TERM where we need different behavior,
we can hardcode that into fish.
* Allow to override this with `fish_features=no-ignore-terminfo fish`
Not sure if we should document this, since it's supposed to be removed soon,
and if someone needs this (which we don't expect), we'd like to know.
* This is supported on a best-effort basis; it doesn't match the previous
behavior exactly. For simplicity of implementation, it will not change
the fact that we now:
* use parm_left_cursor (CSI Ps D) instead of cursor_left (CSI D) if
terminfo claims the former is supported
* no longer support eat_newline_glitch, which seems no longer present
on today's ConEmu and ConHost
* Tested as described in https://github.com/fish-shell/fish-shell/pull/11345#discussion_r2030121580
* add `man fish-terminal-compatibility` to state our assumptions.
This could help terminal emulator developers.
* assume `parm_up_cursor` is supported if the terminal supports XTGETTCAP
* Extract all control sequences to src/terminal_command.rs.
* Remove the "\x1b(B" prefix from EXIT_ATTRIBUTE_MODE. I doubt it's really
needed.
* assume it's generally okay to output 256 colors
Things have improved since commit 3669805627 (Improve compatibility with
0-16 color terminals., 2016-07-21).
Apparently almost every actively developed terminal supports it, including
Terminal.app and GNU screen.
* That is, we default `fish_term256` to true and keep it only as a way to
opt out of the the full 256 palette (e.g. switching to the 16-color
palette).
* `TERM=xterm-16color` has the same opt-out effect.
* `TERM` is generally ignored but add back basic compatiblity by turning
off color for "ansi-m", "linux-m" and "xterm-mono"; these are probably
not set accidentally.
* Since `TERM` is (mostly) ignored, we don't need the magic "xterm" in
tests. Unset it instead.
* Note that our pexpect tests used a dumb terminal because:
1. it makes fish do a full redraw of the commandline everytime, making it
easier to write assertions.
2. it disables all control sequences for colors, etc, which we usually
don't want to test explicitly.
I don't think TERM=dumb has any other use, so it would be better
to print escape sequences unconditionally, and strip them in
the test driver (leaving this for later, since it's a bit more involved).
Closes#11344Closes#11345
I don't think we want to support terminals that implement XTGETTCAP but for
some reason don't use CSI Ps S for scroll forward; that would be a needless
complication.
Let's make ctrl-l / scrollback-push fail hard if a terminal does this.
Confusingly kitty and foot use different response formats, but happily we
no longer care.
An upcoming commit will document that we require the CSI Ps S style.
We canonicalize "ctrl-shift-i" to "ctrl-I".
Both when deciphering this notation (as given to builtin bind),
and when receiving it as a key event ("\e[105;73;6u")
This has problems:
A. Our bind notation canonicalization only works for 26 English letters.
For example, "ctrl-shift-ä" is not supported -- only "ctrl-Ä" is.
We could try to fix that but this depends on the keyboard layout.
For example "bind alt-shift-=" and "bind alt-+" are equivalent on a "us"
layout but not on a "de" layout.
B. While capslock is on, the key event won't include a shifted key ("73" here).
This is due a quirk in the kitty keyboard protocol[^1]. This means that
fish_key_reader's canonicalization doesn't work (unless we call toupper()
ourselves).
I think we want to support both notations.
It's recommended to match all of these (in this order) when pressing
"ctrl-shift-i".
1. bind ctrl-shift-i do-something
2. bind ctrl-shift-I do-something
3. bind ctrl-I do-something
4. bind ctrl-i do-something
Support 1 and 3 for now, allowing both bindings to coexist. No priorities
for now. This solves problem A, and -- if we take care to use the explicit
shift notation -- problem B.
For keys that are not affected by capslock, problem B does not apply. In this
case, recommend the shifted notation ("alt-+" instead of "alt-shift-=")
since that seems more intuitive.
Though if we prioritized "alt-shift-=" over "alt-+" as per the recommendation,
that's an argument against the shifted key.
Example output for some key events:
$ fish_key_reader -cV
# decoded from: \e\[61:43\;4u
bind alt-+ 'do something' # recommended notation
bind alt-shift-= 'do something'
# decoded from: \e\[61:43\;68u
bind alt-+ 'do something' # recommended notation
bind alt-shift-= 'do something'
# decoded from: \e\[105:73\;6u
bind ctrl-I 'do something'
bind ctrl-shift-i 'do something' # recommended notation
# decoded from: \e\[105\;70u
bind ctrl-shift-i 'do something'
Due to the capslock quirk, the last one has only one matching representation
since there is no shifted key. We could decide to match ctrl-shift-i events
(that don't have a shifted key) to ctrl-I bindings (for ASCII letters), as
before this patch. But that case is very rare, it should only happen when
capslock is on, so it's probably not even a breaking change.
The other way round is supported -- we do match ctrl-I events (typically
with shifted key) to ctrl-shift-i bindings (but only for ASCII letters).
This is mainly for backwards compatibility.
Also note that, bindings without other modifiers currently need to use the
shifted key (like "Ä", not "shift-ä"), since we still get a legacy encoding,
until we request "Report all keys as escape codes".
[^1]: <https://github.com/kovidgoyal/kitty/issues/8493>
While at it, don't escape "?", I don't know why 68e167d576 (f-k-r should
use the user's locale, 2016-06-29) did that. Question mark is only special
in combination with redirections.
This add two commands history-last-token-search-backward and
history-last-token-search-forward which behaves like bash's yank-last-arg. So
similar to history-token-search-* but only considers the last argument for
each command.
Closes#10756Closes#11258
As reported on gitter, fish running inside a qemu console randomly fails to
recognize multi-byte sequences like "\e[D" (right); it sometimes recognizes
the first two bytes as "alt-[" and the last byte as the "D" key.
This because 8bf8b10f68 (Extended & human-friendly keys, 2024-03-30) changed
our approach to reading multi-byte key sequences. Previously, we'd wait
forever (or rather fish_sequence_key_delay_ms) for the "D" byte.
As of 8bf8b10f68, we assume the entire sequence is already present in the
input buffer; and stop parsing the sequence if stdin is not readable.
It would be more technically correct to implement the VT state machine but
then we'd probably want to to figure out a timeout or a reset key, in case
of transport or terminal issues.
Returning early is also what we have historically done for multi-byte code
points. Also, other terminal programs have been using it for many years
without problems.
I don't know why this happens in qemu but it seems we can work around by
setting a 1ms timeout. This timeout should be small enough two keys "escape"
and "[" typed by a human will still be seen separate.
Refs:
https://matrix.to/#/!YLTeaulxSDauOOxBoR:matrix.org/$Cfi9wL8FGLAI6_VAQWG2mG_VxsADUPvdPB46P41Jdbshttps://matrix.to/#/!YLTeaulxSDauOOxBoR:matrix.org/$O_-LZ1W7Dk6L_4Rj0MyCry6GtO2JQlEas8fH9PrSYT8
As reported in b5736c5535 (Extend iTerm CSI u workaround to < 3.5.12,
2025-02-20), iTerm 3.5.12 has resolved our issues related to the kitty
keyboard protocol. Enable it here too, matching the release branch.
The flag to gate this is set for versions of iTerm that don't have sufficient
support for the kitty keyboard protocol. CSI u is (more or less) the encoding
used by that protocol. Let's name things accordingly. My bad.
Whenever we add logic to print a control sequence that we hadn't printed
before, there is a nonzero risk that a terminal mishandles it.
Terminal-specific workarounds cause pain but are probably better than not
being able to use any new commands provided by terminals.
There is no universal way to identify a terminal. Device attributes (primary
through tertiary) typically get spoofed responses, likely not good enough
for working around bugs in specific versions of a terminal.
The de-facto standard for the terminal name and version is XTVERSION.
It's usually specific to the terminal, except for something like VTE-based
terminals, where we get this (which seems good enough also)
printf '\x1b[>0q'; cat
^[P>|VTE(7803)^[\
Of course querying for XTVERSION can trigger terminal bugs just as well. Let's
start querying for it now -- even without a concrete use case -- to increase
the chance we can use it during crunch time when we don't want to test
anymore. (We typically discover buggy terminals only very late in the release
cycle, most prominently after a release).
When we enable/disable terminal protocols,
we use atomic operations because of issues like
1. halfway through enabling, we might be interrupted by a signal handler.
2. our SIGTERM handler runs the (idempotent) disabling sequences,
so the operations must be async-signal safe.
The flags to keep track of whether things like kitty keyboard protocol are enabled
are "mirrored" between the enabling and disabling logic:
- the enabling logic marks it as enabled *before* enabling anything
- the disabling logic marks it as disabled *after* everything has been disabled
This ensures that we are well-behaved in issue 1; we will always (perhaps
redundantly) disable the kitty keyboard protocol.
We forgot to use the same ordering for bracketed paste.
If we get SIGTERM after this line
BRACKETED_PASTE.store(false, Ordering::Release);
we might exit with bracketed paste still turned on.
e697add5b5 (Feature flag to prevent executing off buffered keys, 2025-01-02)
breaks my expectations/habits, and it breaks Midnight Commander.
Additionally, I'm not aware of any case where it actually adds security.
We generally assume that terminal echoback sequences do not contain
control characters except for well-known escape sequences.
This backs out commit e697add5b5.
See #10987, #10991
Every reader gets their own wait handle which is wrong and not actually
needed - it's a singleton. We should probaly make it global. Let's
do an intermediate solution for now -- not much time this weekend ;).
Fixes#11110
The two terminals Midnight Commander and dvtm are special in that
they filter requests (or perhaps responses) like
printf "\x1b[0c"
and don't implement the response themselves -- so we never get
one. Let's work around that until we can fix it.
Disable the kitty protocol in mc for now (to keep the code simple),
though we could certainly re-enable it.
Fixes 64859fc242 (Blocking wait for responses to startup queries, 2025-01-25).
This was copied from C++ code but we have overflow checks, which
forces us to actually handle errors.
While at it, add some basic error logging.
Fixes#11092
Today we might
1. enable modifyOtherKeys
2. get a reply indicating the kitty keyboard protocol is supported
3. because of 2, we never turn off modifyOtherKeys again
Let's get rid of this weird issue by enabling either modifyOtherKeys
or the kitty enhancements only after we know whether the kitty protocol
is supported.
This means we need to call terminal_protocols_enable_ifn() before every
call to readch() until the querying is done. Fortunately, this is
already in place in read_normal_chars(); there are other places that
call readch() but none of those is executed until querying has completed.
At startup we query for
- the cursor position (CSI 6 n)
- kitty keyboard protocol support (CSI ? u)
- terminfo capabilities via XTGETTCAP
Since we don't wait for responses, those can leak into child processes.
Some child processes like fzf cannot decode DCS replies. Plug the
leak by ending each round of querying by asking for the Primary Device
Attribute, and resume input processing only after a response has been
received, (or ctrl-c as an escape hatch).
This is a nice simplification. Tested with the lowest common
denominator (putty, Terminal.app and st).
Fixes#11079
The st terminal wrongly parses CSI ? u as DECRC. A fix has been
proposed upstream. Let's also work around it I guess (not to mention
that querying in the first place is also sort of a workaround).
This reverts commit ebdc3a0393.
Not discussed, includes a new thing that queries the terminal for the client OS
when what is really needed is just a `uname` - which would also work on Terminal.app.
As mentioned in
https://github.com/fish-shell/fish-shell/pull/11045#discussion_r1915994998,
we need to refresh TTY timestamps to avoid timing-based issues.
For some context see
git log --grep='[Rr]efresh.* TTY'
Make things more consistent again. I don't know if all of these are
absolutely necessary, hoping to find out later (and consolidate this
logic in outputter).
And leave the old behavior under the name "cancel-commandline".
This renames "cancel-commandline-traditional" back to
"cancel-commandline", so the old name triggers the old behavior.
Fixes#10935
Zellij 0.41.2 has a bug where it responds to
printf '\x1b[?2026$p'; cat -v
with '^[[2026;2$y' (DECRQM) instead of '^[[?2026;2$y' (DECRPM).
This is fixed by https://github.com/zellij-org/zellij/pull/3884
We fail to parse it, leading to an extra y added to the input queue.
Since it seems easy to work around for us, let's do that, I guess.
Some terminals such as conhost and putty cannot parse DCS commands,
and will echo them back.
Work around this by making sure that this echoed text will not
be visible.
Do so by temporarily enabling the alternative screen buffer when
sending DCS queries (in this case only XTGETTCAP). The alternative
screen buffer feature seems widely supported, and easier to get right
than trying to clear individual lines etc.
The alternative screen may still be visible for a
short time. Luckily we can use [Synchronized Output](
https://gist.github.com/christianparpart/d8a62cc1ab659194337d73e399004036)
to make sure the screen change is never visible to the user.
Querying support for that is deemed safe since it only requires a
CSI command.
Note that it seems that every terminal that supports Synchronized
Output also parses DCS commands successfully. This means that we
could get away without the alternative screen buffer in practice.
Not sure yet.
The implementation is slightly more complex than necessary in that it
defines a redundant ImplicitEvent. This is for two reasons: 1. I have
a pending change that wants to use it, so this removes diff noise and
2. we historically have sc/input_common.rs not depend on src/output.rs.
I dont' think any are strong reasons though.
If I run "sleep 3", type a command and hit enter, then there is no
obvious way to cancel or edit the imminent command other than ctrl-c
but that also cancels sleep, and doesn't allow editing. (ctrl-z sort
of works, but also doesn't allow editing).
Let's try to limit ourselves to inserting the buffered command
(translating enter to a newline), and only execute once the user
actually presses enter after the previous command is done.
Hide it behind a new feature flag for now.
By making things less scary, this might be more user-friendly, at
the risk of breaking expectations in some cases.
This also fixes a class of security issues where a command like
`cat malicious-file.txt` might output escape sequences, causing
the terminal to echo back a malicious command; such files can still
insert into the command line but at least not execute it directly.
(Since it's only fixed partially I'm not really sure if the security
issue is a good enough motivation for this particular change.)
Note that bracketed paste probably has similar motivation as this feature.
Part of #10987Closes#10991
Some terminals like the Linux console don't support indn (scroll
forward). Let's query for the presence of these features, and fall
back to the traditional behavior if absent.
For now, break with the tradition of using the terminfo database that
we read ourselves. Instead ask the terminal directly via XTGETTCAP.
This is a fairly young feature implemented by terminals like xterm,
foot and kitty, however xterm doesn't expose these capabilities at
this point.
This is a good opportunity to try XTGETTCAP, since these are
capabilities we haven't used before. Advantages of XTGETTCAP are that
it works across SSH and is independent of $TERM (of course ignoring
$TERM may also be breaking to some users). Let's see if it sees
adoption in practice.
Tested to work on foot and kitty, allowing the default ctrl-l binding
to work without erasing any screen content.
See #11003
The new ctrl-l implementation relies on Cursor Position Reporting (CPR)
This may not work on exotic terminals that don't support CSI 6n yet
As a workaround, probe for this feature by sending a CSI 6n (CPR)
on startup. Until the terminal responds, have scrollback-push fall
back to clear-screen.
The theoretical problem here is that we might handle scrollback-push
before we have handled the response to our feature probe. That seems
fairly unlikely; also e49dde87cc has the same characteristics.
This could query a capability instead (via XTGETTCAP or otherwise)
but I haven't found one; and this seems at least as reliable.
While at it, change the naming a bit.
See #11003
After we query kitty keyboard protocol support,
we send CSI 5n, to also receive a response if
the protocol is not supported.
However we don't bother to wait for the response, so this extra
message is not really useful (only to get better logs). Remove it.
With tmux 3.0 (from 2019) inside SSH, the CSI 5n response is echoed.
I guess with all other terminals we were just lucky. Move it to
right after where we disable ECHO I guess.
In general, asynchronous requests create a lot of potential for error,
we should try to get away from them.
These aren't typically used in the terminal but they are present on
many keyboards.
Also reorganize the named key constants a bit. Between F500 and
ENCODE_DIRECT_BASE (F600) we have space for 256 named keys.
The FdReadableSet api was always intended to be converted to use Duration
instead of usec/msec once the ffi was removed. This lets us be explicit about
forever/infinite timeouts and removes the (small) chance of a collision between
u64::MAX and INFINITE.
I tried this out with `type Timeout = Option<Duration>` (only without the alias)
but was unhappy with easy it is to accidentally use `None` when you meant a
timeout of zero.
On ctrl-l we send `\e[2J` (Erase in Display). Some terminals interpret
this to scroll the screen content instead of clearing it. This happens
on VTE-based terminals like gnome-terminal for example.
The traditional behavior of ctrl-l erasing the screen (but not the
rest of the scrollback) is weird because:
1. `ctrl-l` is the easiest and most portable way to push the prompt
to the top (and repaint after glitches I guess). But it's also a
destructive action, truncating scrollback. I use it for scrolling
and am frequently surprised when my scroll back is missing
information.
2. the amount of lines erased depends on the window size.
It would be more intuitive to erase by prompts, or erase the text
in the terminal selection.
Let's use scrolling behavior on all terminals.
The new command could also be named "push-to-scrollback", for
consistency with others. But if we anticipate a want to add other
scrollback-related commands, "scrollback-push" is better.
This causes tests/checks/tmux-history-search.fish to fail; that test
seems pretty broken; M-d (alt-d) is supposed to delete the current
search match but there is a rogue "echo" that is supposed to invalidate
the search match. I'm not sure how that ever worked.
Also, pexepect doesn't seem to support cursor position reporting,
so work around that.
Ref: https://codeberg.org/dnkl/foot/wiki#how-do-i-make-ctrl-l-scroll-the-content-instead-of-erasing-it
as of wiki commit b57489e298f95d037fdf34da00ea60a5e8eafd6d
Closes#10934
We parse "\e\e[x" as alt-modified "Invalid" key. Due to this extra
modifier, we accidentally add it to the input queue, instead of
dropping this invalid key.
We don't really want to try to extract some valid keys from this
invalid sequence, see also the parent commit.
This allows us to remove misplaced validation that was added by
e8e91c97a6 (fish_key_reader: ignore sentinel key, 2024-04-02) but
later obsoleted by 66c6e89f98 (Don't add collateral sentinel key to
input queue, 2024-04-03).
This situation can be triggered in practice inside a terminal like tmux
3.5 by running
tmux new-session fish -C 'sleep 2' -d reader -o log-file
and typing "alt-escape x"
The log shows that we drop treat this as alt-[ and drop the x on the floor.
reader: Read char alt-\[ -- Key { modifiers: Modifiers { ctrl: false,
alt: true, shift: false }, codepoint: '[' } -- [27, 91, 120]
This input ("\e[x") is ambiguous.
It looks like it could mean "alt-[,x". However that conflicts with a
potential future CSI code, so it makes no sense to try to support this.
Returning "None" from parse_csi() causes this weird behavior of
returning "alt-[" and dropping the rest of the parsed sequence.
This is too easy; it has even crept into a bunch of places
where the input sequence is actually valid like "VT200 button released"
but where we definitely don't want to report any key.
Fix the default: report no key for all unknown sequences and
intentionally-suppressed sequences. Treat it at "alt-[" only when
there is no input byte available, which is more or less unambiguous,
hence a strong enough signal that this is a actually "alt-[".
When the user clicks somewhere in the prompt, kitty asks the shell
to move the cursor there (since there is not much else to do).
This is currently implemented by sending an array of
forward-char-passive commands. This has problems, for example it
is really slow on large command lines (probably because we repaint
everytime).
Implement kitty's `click_events=1` flag to set the
position directly. To convert from terminal-coordinates
to fish-coordinates, query [CSI 6 n Report Cursor
Position](https://invisible-island.net/xterm/ctlseqs/ctlseqs.html)
and use it to compute the left prompt's terminal-coordinates (which
are (0, 0) in fish-coordinates).
Unfortunately this doesn't yet work correctly while the terminal
is scrolled. This is probably because the cursor position is wrong
if off-screen. To fix that we could probably record the cursor
position while not scrolled, but it doesn't seem terribly important
(the existing implementation also doesn't get it right).
We still turn off mouse reporting. If we turned it on, it
would be harder to select text in the terminal itself (not fish).
This would typically mean that mouse-drag will alter fish's
selection and shift+mouse-drag or alt+mouse-drag can be used.
To improve this, we could try to synchronize the selection: if parts
of the fish commandline are selected in the terminal's selection,
copy that to fish's selection and vice versa.
Or maybe there is an intuitive criteria, like: whenever we receive a
mouse event outside fish, turn off mouse reporting, and turn it back on
whenver we receive new keyboard input. One problem is that we lose
one event (though we could send it back to the terminal). Another
problem is we would turn it back on too late in some scenarios.
Closes#10932
