From a6674483c17f1676fffb003c7919c42ed465ce88 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sat, 26 Dec 2020 19:33:54 +0100 Subject: [PATCH 001/105] CHANGELOG: Add more issues to 3.2 Importantly I had added some of the `math` things to 3.1 by accident, this movs them to 3.2 --- CHANGELOG.rst | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 7accb82ac..c62fe8ce5 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -80,7 +80,7 @@ Scripting improvements file descriptor. This allows better error recovery and is more in line with other shells' behaviour (:issue:`7038`). - ``jobs --quiet PID`` no longer prints "no suitable job" if the job for PID does not exist (eg because it has finished) (:issue:`6809`). -- All builtins that query if something exists now take ``--query`` as the long form for ``-q``. ``--quiet`` is deprecated for ``command``, ``jobs`` and ``type`` (:issue:`7276`). +- All builtins that query if something exists now take ` ``--quiet`` is deprecated for ``command``, ``jobs`` and ``type`` (:issue:`7276`). - ``argparse`` now only prints a backtrace with invalid options to argparse itself (:issue:`6703`). - ``complete`` takes the first argument as the name of the command if the ``--command``/``-c`` option is not used (``complete git`` is treated like ``complete --command git``), and can show the loaded completions for specific commands with ``complete COMMANDNAME`` (:issue:`7321`). - ``set_color -b`` (without an argument) no longer prints an error message, matching other invalid invocations of this command (:issue:`7154`). @@ -94,9 +94,12 @@ Scripting improvements - ``test -t``, for testing whether file descriptors are connected to a terminal, works for file descriptors 0, 1, and 2 (:issue:`4766`). It can still return incorrect results in other cases (:issue:`1228`). - Trying to run scripts with Windows line endings (CRLF) via the shebang produces a sensible error (:issue:`2783`). - An ``alias`` that delegates to a command with the same name no longer triggers an error about recursive completion (:issue:`7389`). -- ``math`` now has a ``--base`` option to output the result in hexadecimal or octal (:issue:`7496`). +- ``math`` now has a ``--base`` option to output the result in hexadecimal or octal (:issue:`7496`) and some more specific errors (:issue:`7508`). +- ``math`` learned bitwise functions ``bitand``, ``bitor`` and ``bitxor``, used like ``math "bitand(0xFE, 5)"`` (:issue:`7281`). +- ``math`` learned tau for those wishing to cut down on typing "2 * pi". - ``string`` subcommands now quit early when used with ``--quiet`` (:issue:`7495`). - Failed redirections will now set ``$status`` (:issue:`7540`). +- ``read`` can now read interactively from other files, so e.g. forcing it to read from the terminal via ``read Date: Sat, 26 Dec 2020 19:36:55 +0100 Subject: [PATCH 002/105] Increase issue lock time to half a year Sometimes three months is quite soon, let's see how half a year works out. --- .github/workflows/lockthreads.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/lockthreads.yml b/.github/workflows/lockthreads.yml index 33784c01e..60699703a 100644 --- a/.github/workflows/lockthreads.yml +++ b/.github/workflows/lockthreads.yml @@ -11,6 +11,6 @@ jobs: - uses: dessant/lock-threads@v2 with: github-token: ${{ github.token }} - issue-lock-inactive-days: '90' - pr-lock-inactive-days: '90' + issue-lock-inactive-days: '180' + pr-lock-inactive-days: '180' issue-exclude-labels: 'question' From d3de09da83c1881c76abd3b9496a8901bbf5b2c8 Mon Sep 17 00:00:00 2001 From: Nunzarius Date: Sat, 26 Dec 2020 11:21:13 -0500 Subject: [PATCH 003/105] Added completions for ldapsearch --- share/completions/ldapsearch.fish | 45 +++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 share/completions/ldapsearch.fish diff --git a/share/completions/ldapsearch.fish b/share/completions/ldapsearch.fish new file mode 100644 index 000000000..75e5fc552 --- /dev/null +++ b/share/completions/ldapsearch.fish @@ -0,0 +1,45 @@ +complete -c ldapsearch -s V -d 'Print version info' +complete -c ldapsearch -o VV -d 'Print version info and exit' +complete -c ldapsearch -s d -x -d 'Set the LDAP debug level' +complete -c ldapsearch -s n -d 'Show what would be done, but don\'t actually perform search' +complete -c ldapsearch -s v -d 'Run in verbose mode' +complete -c ldapsearch -s c -d 'Continuous operation mode' +complete -c ldapsearch -s u -d 'Include User Friendly Name of the Distinguished Name (DN)' +complete -c ldapsearch -s t -d 'Write retrieved non-printable values to temp files' +complete -o ldapsearch -o tt -d 'Write all retrieved values to temp files' +complete -c ldapsearch -s T -r -d 'Write temp files to specified directory' +complete -c ldapsearch -s F -x -d 'URL prefix for temp files' +complete -c ldapsearch -s A -d 'Retrieve attributes only' +complete -c ldapsearch -s L -d 'Search results are display in LDAP Data Interchange Format' +complete -c ldapsearch -s S -x -d 'Sort the entries returned based on attribute' +complete -c ldapsearch -s b -x -d 'Specify starting point for the search' +complete -c ldapsearch -s s -xa 'base one sub children' -d 'Specify the scope of the search' +complete -c ldapsearch -s a -xa 'never always search find' -d 'Specify how aliases dereferencing is done' +complete -c ldapsearch -s l -x -d 'Set max number of seconds for a search to complete' +complete -c ldapsearch -s z -x -d 'Set max number of entries for a search' +complete -c ldapsearch -s f -r -d 'Read lines from file, performing one search for each' +complete -c ldapsearch -s M -d 'Enable manage DSA IT control' +complete -c ldapsearch -o MM -d 'Enable manage DSA IT with critical control' +complete -c ldapsearch -s x -d 'Use simple authentication instead of SASL' +complete -c ldapsearch -s D -x -d 'Specify Distinguished Name to bind to the LDAP directory' +complete -c ldapsearch -s W -d 'Prompt for simple authentication' +complete -c ldapsearch -s w -x -d 'Set password for simple authentication' +complete -c ldapsearch -s y -r -d 'Use contents of file as password for simple authentication' +complete -c ldapsearch -s H -x -d 'Specify URI(s) referring to the ldap server(s)' +complete -c ldapsearch -s h -x -d 'Specify an alternate host' +complete -c ldapsearch -s p -x -d 'Specify an alternate TCP port' +complete -c ldapsearch -s P -xa '2 3' -d 'Specify the LDAP protocol version' +complete -c ldapsearch -s e -x -d 'Specify general extensions' +complete -c ldapsearch -s E -x -d 'Specify search extensions' +complete -c ldapsearch -s o -x -d 'Specify general options' +complete -c ldapsearch -s O -x -d 'Specify SASL security properties' +complete -c ldapsearch -s I -d 'Enable SASL Interactive mode' +complete -c ldapsearch -s Q -d 'Enable SASL Quiet mode' +complete -c ldapsearch -s N -d 'Do not use reverse DNS to canonicalize SASL host name' +complete -c ldapsearch -s U -x -d 'Specify the authentication ID for SASL bind' +complete -c ldapsearch -s R -x -d 'Specify the realm of authentication ID for SASL bind' +complete -c ldapsearch -s X -x -d 'Specify the requested authorization ID for SASL bind' +complete -c ldapsearch -s Y -x -d 'Specify the SASL mechanism to be used for authentication' +complete -c ldapsearch -s Z -d 'Issue StartTLS extended operation' +complete -c ldapsearch -o ZZ -d 'Issue StartTLS ectended operation only if succesful' + From 315f8f8a8307fa41a2594162b1bc3864dc6c6272 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Sat, 26 Dec 2020 12:16:00 -0800 Subject: [PATCH 004/105] Relnote ldapsearch completions [ci skip] --- CHANGELOG.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index c62fe8ce5..aee30b516 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -231,6 +231,7 @@ Completions - ``imv`` (:issue:`6675`) - ``julia`` (:issue:`7468`) - ``k3d`` (:issue:`7202`) + - ``ldapsearch`` (:issue:`7578`) - ``micro`` (:issue:`7339`) - ``mpc`` (:issue:`7169`) - Metasploit's ``msfconsole``, ``msfdb`` and ``msfvenom`` (:issue:`6930`) From 94d18c1ac5825123bcf57b5cd0483fca8c92b4d7 Mon Sep 17 00:00:00 2001 From: Ilan Cosman Date: Sat, 26 Dec 2020 14:13:08 -0800 Subject: [PATCH 005/105] CHANGELOG: Add missing --query --- CHANGELOG.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index aee30b516..25c6b8a5e 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -80,7 +80,7 @@ Scripting improvements file descriptor. This allows better error recovery and is more in line with other shells' behaviour (:issue:`7038`). - ``jobs --quiet PID`` no longer prints "no suitable job" if the job for PID does not exist (eg because it has finished) (:issue:`6809`). -- All builtins that query if something exists now take ` ``--quiet`` is deprecated for ``command``, ``jobs`` and ``type`` (:issue:`7276`). +- All builtins that query if something exists now take ``--query``. ``--quiet`` is deprecated for ``command``, ``jobs`` and ``type`` (:issue:`7276`). - ``argparse`` now only prints a backtrace with invalid options to argparse itself (:issue:`6703`). - ``complete`` takes the first argument as the name of the command if the ``--command``/``-c`` option is not used (``complete git`` is treated like ``complete --command git``), and can show the loaded completions for specific commands with ``complete COMMANDNAME`` (:issue:`7321`). - ``set_color -b`` (without an argument) no longer prints an error message, matching other invalid invocations of this command (:issue:`7154`). From bc91a13ba317d91c385a17c66c88f3507af85079 Mon Sep 17 00:00:00 2001 From: Matthew Dutson Date: Sat, 26 Dec 2020 14:45:48 -0700 Subject: [PATCH 006/105] Revise through "Quotes" section --- doc_src/index.rst | 117 ++++++++++++++++++++-------------------------- 1 file changed, 50 insertions(+), 67 deletions(-) diff --git a/doc_src/index.rst b/doc_src/index.rst index f3f70deb9..ae82f301d 100644 --- a/doc_src/index.rst +++ b/doc_src/index.rst @@ -4,84 +4,76 @@ Introduction ============ -This is the documentation for *fish*, the **f**\ riendly **i**\ nteractive **sh**\ ell. +This is the documentation for **fish**, the **f**\ riendly **i**\ nteractive **sh**\ ell. -A shell is a program which helps you operate your computer by starting other programs. fish offers a command-line interface focused on usability and interactive use. +A shell is a program that helps you operate your computer by starting other programs. fish offers a command-line interface focused on usability and interactive use. Unlike other shells, fish does not follow the POSIX standard, but still roughly belongs to the same family. Some of the special features of fish are: -- **Extensive UI**: `syntax highlighting`_, autosuggestions_, `tab completion`_ and selection lists that can be navigated and filtered. +- **Extensive UI**: `Syntax highlighting`_, autosuggestions_, `tab completion`_ and selection lists that can be navigated and filtered. - **No configuration needed**: fish is designed to be ready to use immediately, without requiring extensive configuration. -- **Easy scripting**: new functions_ can be added on the fly. The syntax is easy to learn and use. +- **Easy scripting**: New functions_ can be added on the fly. The syntax is easy to learn and use. -This bit of the documentation is a quick guide on how to get going. If you are new to this, see the :ref:`tutorial `. +This page gives an overview of fish's features, syntax, and interface. If this is your first time using fish, see the :ref:`tutorial `. Installation and Start ====================== -This section is on how to install, uninstall, start and exit a fish shell and on how to make fish the default shell: +This section describes how to install, uninstall, start, and exit the fish shell. It also explains how to make fish the default shell: - `Installation`_: How to install fish - `Starting and Exiting`_ How to start and exit a fish shell - `Executing Bash`_: How to execute bash commands in fish -- `Default Shell`_: How to switch to fish as the default shell +- `Default Shell`_: How to make fish the default shell - `Uninstalling`_: How to uninstall fish - Installation ------------ Up-to-date instructions for installing the latest version of fish are on the `fish homepage `_. -To install the development version of fish see the instructions at the `project's GitHub page `_. - +To install the development version of fish, see the instructions on the `project's GitHub page `_. Starting and Exiting -------------------- Once fish has been installed, open a terminal. If fish is not the default shell: -- Enter ``fish`` to start a fish shell:: +- Type ``fish`` to start a fish shell:: > fish - -- Enter ``exit`` to exit a fish shell:: +- Type ``exit`` to exit a fish shell:: > exit - Executing Bash -------------- -If fish is your default shell and you want to copy commands from the internet that are written in a different shell language, bash for example, you can proceed in the following way: +If fish is your default shell and you want to copy commands from the internet that are written in bash (the default shell on most systems), you can proceed in one of the following two ways: -Consider that ``bash`` is also a command. With ``man bash`` you can see that there are two ways to do this: - -- ``bash`` has a switch ``-c`` to read from a string:: +- Use the ``bash`` command with the ``-c`` switch to read from a string:: > bash -c 'some bash command' - -- ``bash`` without a switch opens a bash shell that you can use and ``exit`` afterwards:: +- Use ``bash`` without a switch to open a bash shell you can use and ``exit`` afterward:: > bash $ some bash command $ exit > _ - Default Shell ------------- -You can make fish your default shell by adding fish's executable in two places: +To make fish your default shell: -- add ``/usr/local/bin/fish`` to ``/etc/shells`` -- change your default shell with ``chsh -s`` to ``/usr/local/bin/fish`` +- Add the line ``/usr/local/bin/fish`` to ``/etc/shells``. +- Change your default shell with ``chsh -s /usr/local/bin/fish``. For detailed instructions see :ref:`Switching to fish `. @@ -93,87 +85,80 @@ For uninstalling fish: see :ref:`FAQ: Uninstalling fish `. Shebang Line ------------ -Since scripts for shell commands can be written in many different languages, they need to carry information about what interpreter is needed to execute them. For this they are expected to have a first line, the shebang line, which names an executable for this purpose. +Because shell scripts are written in many different languages, they need to carry information about which interpreter should be used to execute them. For this, they are expected to have a first line, the shebang line, which names the interpreter executable. A script written in ``bash`` would need a first line like this:: #!/bin/bash +This tells the shell to execute the file with the interpreter located at ``/bin/bash``. -This line tells the shell to execute the file with the bash interpreter, that is located at the path ``/bin/bash``. +For a script written in another language, just replace ``/bin/bash`` with the interpreter for that language (for example: ``/bin/python`` for a python script, or ``/usr/local/bin/fish`` for a fish script). -For a script written in another language, just replace the interpreter ``/bin/bash`` with the language interpreter of that other language (for example: ``/bin/python`` for a python script, or ``/usr/local/bin/fish`` for a fish script). - -This line is only needed when scripts are executed without specifying the interpreter. For functions inside fish or when executing a script with ``fish /path/to/script`` they aren't required (but don't hurt either!). +This line is only needed when scripts are executed without specifying the interpreter. For functions inside fish or when executing a script with ``fish /path/to/script``, a shebang is not required (but it doesn't hurt!). .. _syntax: -Syntax overview +Syntax Overview =============== -Shells like fish are used by giving them commands. Every fish command follows the same basic syntax. - -A command is executed by writing the name of the command followed by any arguments. - -Example:: +Shells like fish are used by giving them commands. Every fish command follows the same basic syntax. A command is executed by writing the name of the command followed by any arguments. For example:: echo hello world -This calls the :ref:`echo ` command. ``echo`` is a command which will write its arguments to the screen. In the example above, the output will be 'hello world'. Everything in fish is done with commands. There are commands for performing a set of commands multiple times, commands for assigning variables, commands for treating a group of commands as a single command, etc.. And every single command follows the same basic syntax. +This calls the :ref:`echo ` command. ``echo`` writes its arguments to the screen. In the example above, the output is 'hello world'. Everything in fish is done with commands. There are commands for repeating other commands, commands for assigning variables, commands for treating a group of commands as a single command, etc. All of these commands follow the same basic syntax. -If you want to find out more about the echo command used above, read the manual page for the echo command by writing: ``man echo`` +To learn more about the ``echo`` command, read its manual page by typing ``man echo``. ``man`` is a command for displaying a manual page on a given topic. It takes the name of the manual page to display as an argument. There are manual pages for almost every command. There are also manual pages for many other things, such as system libraries and important files. -``man`` is a command for displaying a manual page on a given topic. The man command takes the name of the manual page to display as an argument. There are manual pages for almost every command on most computers. There are also manual pages for many other things, such as system libraries and important files. - -Every program on your computer can be used as a command in fish. If the program file is located in one of the directories in the PATH_, you can just use the name of the program to use it. Otherwise the whole filename, including the directory (like ``/home/me/code/checkers/checkers`` or ``../checkers``) has to be used. +Every program on your computer can be used as a command in fish. If the program file is located in one of the PATH_ directories, you can just type the name of the program to use it. Otherwise the whole filename, including the directory (like ``/home/me/code/checkers/checkers`` or ``../checkers``) is required. Here is a list of some useful commands: -- :ref:`cd `, change the current directory -- ``ls``, list files and directories -- ``man``, display a manual page on the screen -- ``mv``, move (rename) files -- ``cp``, copy files -- :ref:`open `, open files with the default application associated with each filetype -- ``less``, list the contents of files +- :ref:`cd `: Change the current directory +- ``ls``: List files and directories +- ``man``: Display a manual page +- ``mv``: Move (rename) files +- ``cp``: Copy files +- :ref:`open `: Open files with the default application associated with each filetype +- ``less``: Display the contents of files -Commands and parameters are separated by the space character ``' '``. Every command ends with either a newline (i.e. by pressing the return key) or a semicolon ``;``. More than one command can be written on the same line by separating them with semicolons. +Commands and arguments are separated by the space character ``' '``. Every command ends with either a newline (by pressing the return key) or a semicolon ``;``. Multiple commands can be written on the same line by separating them with semicolons. -A switch is a very common special type of argument. Switches almost always start with one or more hyphens ``-`` and alter the way a command operates. For example, the ``ls`` command usually lists all the files and directories in the current working directory, but by using the ``-l`` switch, the behavior of ``ls`` is changed to not only display the filename, but also the size, permissions, owner and modification time of each file. +A switch is a very common special type of argument. Switches almost always start with one or more hyphens ``-`` and alter the way a command operates. For example, the ``ls`` command usually lists the names of all files and directories in the current working directory. By using the ``-l`` switch, the behavior of ``ls`` is changed to not only display the filename, but also the size, permissions, owner, and modification time of each file. -Switches differ between commands and are documented in the manual page for each command. Some switches are common to most command though, for example ``--help`` will usually display a help text, ``-i`` will often turn on interactive prompting before taking action, while ``-f`` will turn it off. +Switches differ between commands and are usually documented on a command's manual page. There are some switches, however, that are common to most commands. For example, ``--help`` will usually display a help text, ``--version`` will usually display the command version, and ``-i`` will often turn on interactive prompting before taking action. -.. _syntax-words: +.. _terminology: -Some common words ------------------ +Terminology +----------- -This is a short explanation of some of the commonly used words in fish. +Here we define some of the terms used on this page and throughout the rest of the fish documentation: -- **argument** a parameter given to a command +- **Argument**: A parameter given to a command. -- **builtin** a command that is implemented in the shell. Builtins are commands that are so closely tied to the shell that it is impossible to implement them as external commands. +- **Builtin**: A command that is implemented by the shell. Builtins are so closely tied to the operation of the shell that it is impossible to implement them as external commands. -- **command** a program that the shell can run. In another sense also specifically an external command (i.e. neither a function or builtin). +- **Command**: A program that the shell can run. -- **function** a block of commands that can be called as if they were a single command. By using functions, it is possible to string together multiple smaller commands into one more advanced command. +- **Function**: A block of commands that can be called as if they were a single command. By using functions, it is possible to string together multiple simple commands into one more advanced command. -- **job** a running pipeline or command +- **Job**: A running pipeline or command. -- **pipeline** a set of commands stringed together so that the output of one command is the input of the next command +- **Pipeline**: A set of commands strung together so that the output of one command is the input of the next command. -- **redirection** an operation that changes one of the input/output streams associated with a job +- **Redirection**: An operation that changes one of the input or output streams associated with a job. -- **switch** a special flag sent as an argument to a command that will alter the behavior of the command. A switch almost always begins with one or two hyphens. +- **Switch**: A special kind of argument that alters the behavior of a command. A switch almost always begins with one or two hyphens. .. _quotes: Quotes ------ -Sometimes features like `parameter expansion <#expand>`_ and `character escapes <#escapes>`_ get in the way. When that happens, you can use quotes, either ``'`` (single quote) or ``"`` (double quote). Between single quotes, fish will perform no expansions, in double quotes it will only do :ref:`variable expansion `. No other kind of expansion (including :ref:`brace expansion ` and parameter expansion) will take place, the parameter can contain spaces, and escape sequences are ignored. +Sometimes features like `parameter expansion <#expand>`_ and `character escapes <#escapes>`_ get in the way. When that happens, you can use quotes, either single (``'``) or double (``"``). Between single quotes, fish performs no expansions. Between double quotes, fish only performs :ref:`variable expansion `. No other kind of expansion (including :ref:`brace expansion ` or parameter expansion) is performed, and escape sequences (for example, ``\n``) are ignored. Within quotes, whitespace is not used to separate arguments, allowing quoted arguments to contain spaces. -The only backslash escapes that mean anything in single quotes are ``\'``, which escapes a single quote and ``\\``, which escapes the backslash symbol. The only backslash escapes in double quotes are ``\"``, which escapes a double quote, ``\$``, which escapes a dollar character, ``\`` followed by a newline, which deletes the backslash and the newline, and ``\\``, which escapes the backslash symbol. +The only meaningful escape sequences in single quotes are ``\'``, which escapes a single quote and ``\\``, which escapes the backslash symbol. The only meaningful escapes in double quotes are ``\"``, which escapes a double quote, ``\$``, which escapes a dollar character, ``\`` followed by a newline, which deletes the backslash and the newline, and ``\\``, which escapes the backslash symbol. Single quotes have no special meaning within double quotes and vice versa. @@ -187,16 +172,14 @@ removes the file ``cumbersome filename.txt``, while rm cumbersome filename.txt - removes two files, ``cumbersome`` and ``filename.txt``. -:: +Another example:: grep 'enabled)$' foo.txt searches for lines ending in ``enabled)`` in ``foo.txt`` (the ``$`` is special to ``grep``: it matches the end of the line). - .. _escapes: Escaping characters From aaeb7d107c29c3e594eb8ad20a793fd3b7937049 Mon Sep 17 00:00:00 2001 From: elpres Date: Mon, 28 Dec 2020 15:11:53 +0100 Subject: [PATCH 007/105] Fixed sentence in `fish_hg_prompt` docs --- doc_src/cmds/fish_hg_prompt.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc_src/cmds/fish_hg_prompt.rst b/doc_src/cmds/fish_hg_prompt.rst index eb111c2a0..e44a36630 100644 --- a/doc_src/cmds/fish_hg_prompt.rst +++ b/doc_src/cmds/fish_hg_prompt.rst @@ -19,7 +19,7 @@ The fish_hg_prompt function displays information about the current Mercurial rep `Mercurial `_ (``hg``) must be installed. -By default, only the current branch is shown because ``hg status`` can take be slow on large repository. You can enable a more informative prompt by setting the variable ``$fish_prompt_hg_show_informative_status``, for example:: +By default, only the current branch is shown because ``hg status`` can be slow on a large repository. You can enable a more informative prompt by setting the variable ``$fish_prompt_hg_show_informative_status``, for example:: set --universal fish_prompt_hg_show_informative_status From 3c2cf6241bf995fac3de41049106ba74016e2963 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Mon, 28 Dec 2020 22:40:30 +0100 Subject: [PATCH 008/105] Add some error tests for `cd` Makes work on #7577 easier. --- tests/checks/cd.fish | 63 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/tests/checks/cd.fish b/tests/checks/cd.fish index 88cb84da6..b08e7af74 100644 --- a/tests/checks/cd.fish +++ b/tests/checks/cd.fish @@ -126,10 +126,73 @@ test $PWD = $base/-testdir echo $status #CHECK: 0 +# test a few error cases - nonexistent directory +set -l old_cdpath $CDPATH +set -l old_path $PWD +cd nonexistent +#CHECKERR: cd: The directory 'nonexistent' does not exist +#CHECKERR: {{.*}}/cd.fish (line {{\d+}}): +#CHECKERR: builtin cd $argv +#CHECKERR: ^ +#CHECKERR: in function 'cd' with arguments 'nonexistent' +#CHECKERR: called on line {{\d+}} of file {{.*}}/cd.fish + +touch file +cd file +#CHECKERR: cd: 'file' is not a directory +#CHECKERR: {{.*}}/cd.fish (line {{\d+}}): +#CHECKERR: builtin cd $argv +#CHECKERR: ^ +#CHECKERR: in function 'cd' with arguments 'file' +#CHECKERR: called on line {{\d+}} of file {{.*}}/cd.fish + +# a directory that isn't executable +mkdir bad-perms +chmod -x bad-perms +cd bad-perms +#CHECKERR: cd: Permission denied: 'bad-perms' +#CHECKERR: {{.*}}/cd.fish (line {{\d+}}): +#CHECKERR: builtin cd $argv +#CHECKERR: ^ +#CHECKERR: in function 'cd' with arguments 'bad-perms' +#CHECKERR: called on line {{\d+}} of file {{.*}}/cd.fish + +cd $old_path +mkdir -p cdpath-dir/bad-perms +mkdir -p cdpath-dir/nonexistent +mkdir -p cdpath-dir/file +set CDPATH $PWD/cdpath-dir $old_cdpath + +# A different directory with the same name that is first in $CDPATH works. +cd bad-perms +cd $old_path +cd nonexistent +cd $old_path +cd file +cd $old_path + +# Even if the good dirs are later in $CDPATH most errors still aren't a problem +# - they just cause us to keep looking. +cd $old_path +set CDPATH $old_cdpath $PWD/cdpath-dir +cd nonexistent +cd $old_path +cd bad-perms +# Permission errors are still a problem! +#CHECKERR: cd: Permission denied: 'bad-perms' +#CHECKERR: {{.*}}/cd.fish (line {{\d+}}): +#CHECKERR: builtin cd $argv +#CHECKERR: ^ +#CHECKERR: in function 'cd' with arguments 'bad-perms' +#CHECKERR: called on line {{\d+}} of file {{.*}}/cd.fish +cd $old_path +cd file +cd $old_path # cd back before removing the test directory again. cd $oldpwd rm -Rf $base +set -g CDPATH ./ # Verify that PWD on-variable events are sent function __fish_test_changed_pwd --on-variable PWD From df73964ceda6d5aef798157f9a013ca0d439ad90 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Mon, 28 Dec 2020 11:08:54 -0800 Subject: [PATCH 009/105] Clean up some comments around wildcard expansion --- src/wildcard.cpp | 20 +++++++------------- src/wildcard.h | 6 ++---- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/src/wildcard.cpp b/src/wildcard.cpp index 2db059237..14c8ffc2c 100644 --- a/src/wildcard.cpp +++ b/src/wildcard.cpp @@ -867,20 +867,14 @@ void wildcard_expander_t::expand_last_segment(const wcstring &base_dir, DIR *bas /// wrappers around this one. /// /// This function traverses the relevant directory tree looking for matches, and recurses when -/// needed to handle wildcrards spanning multiple components and recursive wildcards. -/// -/// Because this function calls itself recursively with substrings, it's important that the -/// parameters be raw pointers instead of wcstring, which would be too expensive to construct for -/// all substrings. +/// needed to handle wildcards spanning multiple components and recursive wildcards. /// /// Args: /// base_dir: the "working directory" against which the wildcard is to be resolved /// wc: the wildcard string itself, e.g. foo*bar/baz (where * is actually ANY_CHAR) -/// prefix: the string that should be prepended for completions that replace their token. -// This is usually the same thing as the original wildcard, but for fuzzy matching, we -// expand intermediate segments. effective_prefix is always either empty, or ends with a slash -// Note: this is only used when doing completions (for_completions is true), not -// expansions +/// effective_prefix: the string that should be prepended for completions that replace their token. +/// This is usually the same thing as the original wildcard, but for fuzzy matching, we +/// expand intermediate segments. effective_prefix is always either empty, or ends with a slash void wildcard_expander_t::expand(const wcstring &base_dir, const wchar_t *wc, const wcstring &effective_prefix) { assert(wc != nullptr); @@ -890,10 +884,9 @@ void wildcard_expander_t::expand(const wcstring &base_dir, const wchar_t *wc, } // Get the current segment and compute interesting properties about it. - const size_t wc_len = std::wcslen(wc); const wchar_t *const next_slash = std::wcschr(wc, L'/'); const bool is_last_segment = (next_slash == nullptr); - const size_t wc_segment_len = next_slash ? next_slash - wc : wc_len; + const size_t wc_segment_len = next_slash ? next_slash - wc : std::wcslen(wc); const wcstring wc_segment = wcstring(wc, wc_segment_len); const bool segment_has_wildcards = wildcard_has(wc_segment, true /* internal, i.e. look for ANY_CHAR instead of ? */); @@ -949,7 +942,8 @@ void wildcard_expander_t::expand(const wcstring &base_dir, const wchar_t *wc, effective_prefix + wc_segment + L'/'); } - // Recursive wildcards require special handling. + // If we have a recursive wildcard in this segment, we want to recurse into + // subdirectories. size_t asr_idx = wc_segment.find(ANY_STRING_RECURSIVE); if (asr_idx != wcstring::npos) { // Construct a "head + any" wildcard for matching stuff in this directory, and an diff --git a/src/wildcard.h b/src/wildcard.h index 2c1cf055d..4258a70f5 100644 --- a/src/wildcard.h +++ b/src/wildcard.h @@ -24,11 +24,9 @@ enum { /// Expand the wildcard by matching against the filesystem. /// -/// New strings are allocated using malloc and should be freed by the caller. -/// /// wildcard_expand works by dividing the wildcard into segments at each directory boundary. Each -/// segment is processed separatly. All except the last segment are handled by matching the wildcard -/// segment against all subdirectories of matching directories, and recursively calling +/// segment is processed separately. All except the last segment are handled by matching the +/// wildcard segment against all subdirectories of matching directories, and recursively calling /// wildcard_expand for matches. On the last segment, matching is made to any file, and all matches /// are inserted to the list. /// From 6c08141682422f10aa4d88cae4831a3e0613a56f Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Mon, 28 Dec 2020 13:09:28 -0800 Subject: [PATCH 010/105] Add a littlcheck glob test We have some glob tests in fish_tests.cpp, but they are hard to follow. Begin migrating them --- tests/checks/glob.fish | 79 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 tests/checks/glob.fish diff --git a/tests/checks/glob.fish b/tests/checks/glob.fish new file mode 100644 index 000000000..560756833 --- /dev/null +++ b/tests/checks/glob.fish @@ -0,0 +1,79 @@ +# RUN: %fish %s + +cd (mktemp -d) +set tmpdir (pwd -P) + +# Hidden files are only matched with explicit dot. +touch .hidden visible +string join \n * | sort +# CHECK: visible +string join \n .* | sort +# CHECK: .hidden +rm -Rf .hidden visible + +# Trailing slash matches only directories. +touch abc1 +mkdir abc2 +string join \n * | sort +# CHECK: abc1 +# CHECK: abc2 +string join \n */ | sort +# CHECK: abc2/ +rm -Rf * + +# Symlinks are descended into independently. +# Here dir2/link2 is symlinked to dir1/child1. +# The contents of dir2 will be explored twice. +mkdir -p dir1/child1 +touch dir1/child1/anyfile +mkdir dir2 +ln -s ../dir1/child1 dir2/link2 +string join \n **/anyfile | sort +# CHECK: dir1/child1/anyfile +# CHECK: dir2/link2/anyfile + +# But symlink loops only get explored once. +mkdir -p dir1/child2/grandchild1 +touch dir1/child2/grandchild1/differentfile +ln -s ../../child2/grandchild1 dir1/child2/grandchild1/link2 +echo **/differentfile +# CHECK: dir1/child2/grandchild1/differentfile +rm -Rf * + +# Recursive globs handling. +mkdir -p dir_a1/dir_a2/dir_a3 +touch dir_a1/dir_a2/dir_a3/file_a +mkdir -p dir_b1/dir_b2/dir_b3 +touch dir_b1/dir_b2/dir_b3/file_b +string join \n **/file_* | sort +# CHECK: dir_a1/dir_a2/dir_a3/file_a +# CHECK: dir_b1/dir_b2/dir_b3/file_b + +string join \n **a3/file_* | sort +# CHECK: dir_a1/dir_a2/dir_a3/file_a + +string join \n ** | sort +# CHECK: dir_a1 +# CHECK: dir_a1/dir_a2 +# CHECK: dir_a1/dir_a2/dir_a3 +# CHECK: dir_a1/dir_a2/dir_a3/file_a +# CHECK: dir_b1 +# CHECK: dir_b1/dir_b2 +# CHECK: dir_b1/dir_b2/dir_b3 +# CHECK: dir_b1/dir_b2/dir_b3/file_b + +string join \n **/ | sort +# CHECK: dir_a1/ +# CHECK: dir_a1/dir_a2/ +# CHECK: dir_a1/dir_a2/dir_a3/ +# CHECK: dir_b1/ +# CHECK: dir_b1/dir_b2/ +# CHECK: dir_b1/dir_b2/dir_b3/ + +string join \n **a2/** | sort +# CHECK: dir_a1/dir_a2/dir_a3 +# CHECK: dir_a1/dir_a2/dir_a3/file_a + +# Clean up. +cd $HOME +rm -Rf $tmpdir From 43505f7077d1449b5500e46805bcd5fdc2d5e2de Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Mon, 28 Dec 2020 18:11:47 -0800 Subject: [PATCH 011/105] Allow ** glob segments to match zero directories Prior to this change, a glob like `**/file.txt` would only match `file.txt` in subdirectories; the `**` must match at least one directory. This is historical behavior. With this change we move a little closer to bash's implementation by allowing a literal `**` segment to match in the current directory. That is, `**/foo` will match both `foo` and `bar/foo`, while `b**/foo` will only match `bar/foo`. Fixes #7222. --- CHANGELOG.rst | 1 + doc_src/index.rst | 5 ++--- src/wildcard.cpp | 17 +++++++++++++++-- tests/checks/glob.fish | 10 ++++++++++ 4 files changed, 28 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 25c6b8a5e..050fa6c3b 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -45,6 +45,7 @@ Syntax changes and new commands ------------------------------- - Range limits in index range expansions like ``$x[$start..$end]`` may be omitted: ``$start`` and ``$end`` default to 1 and -1 (the last item) respectively. - Logical operators ``&&`` and ``||`` can be followed by newlines before their right operand, matching POSIX shells. +- When globbing, a segment which is exactly ``**`` may now match zero directories. For example ``**/foo`` may match ``foo`` in the current directory (:issue:`7222`). Scripting improvements ---------------------- diff --git a/doc_src/index.rst b/doc_src/index.rst index ae82f301d..dd98587a9 100644 --- a/doc_src/index.rst +++ b/doc_src/index.rst @@ -498,9 +498,9 @@ Wildcards ("Globbing") When a parameter includes an :ref:`unquoted ` ``*`` star (or "asterisk") or a ``?`` question mark, fish uses it as a wildcard to match files. -- ``*`` can match any string of characters not containing ``/``. This includes matching an empty string. +- ``*`` matches any number of characters (including zero) in a file name, not including ``/``. -- ``**`` matches any string of characters. This includes matching an empty string. The matched string can include the ``/`` character; that is, it goes into subdirectories. If a wildcard string with ``**`` contains a ``/``, that ``/`` still needs to be matched. For example, ``**\/*.fish`` won't match ``.fish`` files directly in the PWD, only in subdirectories. In fish you should type ``**.fish`` to match files in the PWD as well as subdirectories. [#]_ +- ``**`` matches any number of characters (including zero), and also descends into subdirectories. If ``**`` is a segment by itself, that segment may match zero times, for compatibility with other shells. - ``?`` can match any single character except ``/``. This is deprecated and can be disabled via the ``qmark-noglob`` :ref:`feature flag`, so ``?`` will just be an ordinary character. @@ -541,7 +541,6 @@ Examples:: end # Lists the .foo files, if any. -.. [#] Unlike other shells, notably zsh. .. [#] Technically, unix allows filenames with newlines, and this splits the ``find`` output on newlines. If you want to avoid that, use find's ``-print0`` option and :ref:`string split0`. .. _expand-command-substitution: diff --git a/src/wildcard.cpp b/src/wildcard.cpp index 14c8ffc2c..f3a411b79 100644 --- a/src/wildcard.cpp +++ b/src/wildcard.cpp @@ -930,6 +930,20 @@ void wildcard_expander_t::expand(const wcstring &base_dir, const wchar_t *wc, } } else { assert(!wc_segment.empty() && (segment_has_wildcards || is_last_segment)); + + if (!is_last_segment && wc_segment == wcstring{ANY_STRING_RECURSIVE}) { + // Hack for #7222. This is an intermediate wc segment that is exactly **. The + // tail matches in subdirectories as normal, but also the current directory. + // That is, '**/bar' may match 'bar' and 'foo/bar'. + // Implement this by matching the wildcard tail only, in this directory. + // Note if the segment is not exactly ANY_STRING_RECURSIVE then the segment may only + // match subdirectories. + this->expand(base_dir, wc_remainder, effective_prefix); + if (interrupted_or_overflowed()) { + return; + } + } + DIR *dir = open_dir(base_dir); if (dir) { if (is_last_segment) { @@ -942,10 +956,9 @@ void wildcard_expander_t::expand(const wcstring &base_dir, const wchar_t *wc, effective_prefix + wc_segment + L'/'); } - // If we have a recursive wildcard in this segment, we want to recurse into - // subdirectories. size_t asr_idx = wc_segment.find(ANY_STRING_RECURSIVE); if (asr_idx != wcstring::npos) { + // Apply the recursive **. // Construct a "head + any" wildcard for matching stuff in this directory, and an // "any + tail" wildcard for matching stuff in subdirectories. Note that the // ANY_STRING_RECURSIVE character is present in both the head and the tail. diff --git a/tests/checks/glob.fish b/tests/checks/glob.fish index 560756833..8777b6117 100644 --- a/tests/checks/glob.fish +++ b/tests/checks/glob.fish @@ -74,6 +74,16 @@ string join \n **a2/** | sort # CHECK: dir_a1/dir_a2/dir_a3 # CHECK: dir_a1/dir_a2/dir_a3/file_a +rm -Rf * + +# Special behavior for #7222. +# The literal segment ** matches in the same directory. +mkdir foo +touch bar foo/bar +string join \n **/bar | sort +# CHECK: bar +# CHECK: foo/bar + # Clean up. cd $HOME rm -Rf $tmpdir From 4c09012b0d8f412f7e23613edcc32979384611d6 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Tue, 29 Dec 2020 12:48:11 +0100 Subject: [PATCH 012/105] tests: Don't rely on $HOME existing Apparently the launchpad tests run with $HOME set to a nonexistent directory. Since we just want *out*, let's just store the previous dir and go back. --- tests/checks/glob.fish | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/checks/glob.fish b/tests/checks/glob.fish index 8777b6117..bb75d23e0 100644 --- a/tests/checks/glob.fish +++ b/tests/checks/glob.fish @@ -1,5 +1,6 @@ # RUN: %fish %s +set -l oldpwd $PWD cd (mktemp -d) set tmpdir (pwd -P) @@ -85,5 +86,5 @@ string join \n **/bar | sort # CHECK: foo/bar # Clean up. -cd $HOME +cd $oldpwd rm -Rf $tmpdir From 84262b126b7a26fb66b00b10a615fe077ea134a2 Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Fri, 25 Dec 2020 07:45:08 +0100 Subject: [PATCH 013/105] build_tools/style.fish: don't format other Python files --- build_tools/style.fish | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/build_tools/style.fish b/build_tools/style.fish index d685c8e44..ea720ecbf 100755 --- a/build_tools/style.fish +++ b/build_tools/style.fish @@ -28,8 +28,8 @@ if test $all = yes exit 1 end set c_files src/*.h src/*.cpp src/*.c - set fish_files (printf '%s\n' share/***.fish) - set python_files **.py + set fish_files share/**.fish + set python_files {doc_src,share,tests}/**.py else # We haven't been asked to reformat all the source. If there are uncommitted changes reformat # those using `git clang-format`. Else reformat the files in the most recent commit. From 85830a5775b872a0b085af0db68fc4519b809f59 Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Thu, 24 Dec 2020 09:42:27 +0100 Subject: [PATCH 014/105] completions/git: don't sort branches and tags This seems a bit more intuitive. --- share/completions/git.fish | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/share/completions/git.fish b/share/completions/git.fish index 14ab29b24..bceedd3fd 100644 --- a/share/completions/git.fish +++ b/share/completions/git.fish @@ -937,14 +937,14 @@ complete -f -c git -n '__fish_git_using_command show' -l oneline -d 'Shorthand f complete -f -c git -n '__fish_git_using_command show' -l encoding -d 'Re-code the commit log message in the encoding' complete -f -c git -n '__fish_git_using_command show' -l expand-tabs -d 'Perform a tab expansion in the log message' complete -f -c git -n '__fish_git_using_command show' -l no-expand-tabs -d 'Do not perform a tab expansion in the log message' -complete -f -c git -n '__fish_git_using_command show' -l notes -a '(__fish_git_refs)' -d 'Show the notes that annotate the commit' +complete -f -c git -n '__fish_git_using_command show' -l notes -k -a '(__fish_git_refs)' -d 'Show the notes that annotate the commit' complete -f -c git -n '__fish_git_using_command show' -l no-notes -d 'Do not show notes' complete -f -c git -n '__fish_git_using_command show' -l show-signature -d 'Check the validity of a signed commit object' ### show-branch complete -f -c git -n __fish_git_needs_command -a show-branch -d 'Shows the commits on branches' -complete -f -c git -n '__fish_git_using_command show-branch' -a '(__fish_git_refs)' -d Rev +complete -f -c git -n '__fish_git_using_command show-branch' -k -a '(__fish_git_refs)' -d Rev # TODO options ### add @@ -1012,7 +1012,7 @@ complete -f -c git -n '__fish_git_using_command branch' -l no-merged -d 'List br ### cherry complete -f -c git -n __fish_git_needs_command -a cherry -d 'Find commits yet to be applied to upstream [upstream [head]]' complete -f -c git -n '__fish_git_using_command cherry' -s v -d 'Show the commit subjects next to the SHA1s' -complete -f -c git -n '__fish_git_using_command cherry' -a '(__fish_git_refs)' -d Upstream +complete -f -c git -n '__fish_git_using_command cherry' -k -a '(__fish_git_refs)' -d Upstream ### cherry-pick complete -f -c git -n __fish_git_needs_command -a cherry-pick -d 'Apply the change introduced by an existing commit' @@ -1090,7 +1090,7 @@ complete -f -c git -n '__fish_git_using_command describe' -l first-parent -d 'Fo ### diff complete -c git -n __fish_git_needs_command -a diff -d 'Show changes between commits, commit and working tree, etc' -complete -c git -n '__fish_git_using_command diff; and not contains -- -- (commandline -opc)' -a '(__fish_git_ranges)' +complete -c git -n '__fish_git_using_command diff; and not contains -- -- (commandline -opc)' -k -a '(__fish_git_ranges)' complete -c git -n '__fish_git_using_command diff' -l cached -d 'Show diff of changes in the index' complete -c git -n '__fish_git_using_command diff' -l staged -d 'Show diff of changes in the index' complete -c git -n '__fish_git_using_command diff' -l no-index -d 'Compare two paths on the filesystem' @@ -1115,7 +1115,7 @@ end ### difftool complete -c git -n __fish_git_needs_command -a difftool -d 'Open diffs in a visual tool' -complete -c git -n '__fish_git_using_command difftool' -a '(__fish_git_ranges)' +complete -c git -n '__fish_git_using_command difftool' -k -a '(__fish_git_ranges)' complete -c git -n '__fish_git_using_command difftool' -l cached -d 'Visually show diff of changes in the index' complete -f -c git -n '__fish_git_using_command difftool' -a '(__fish_git_files modified deleted)' complete -f -c git -n '__fish_git_using_command difftool' -s g -l gui -d 'Use `diff.guitool` instead of `diff.tool`' @@ -1150,7 +1150,7 @@ complete -f -c git -n __fish_git_needs_command -a init -d 'Create an empty git r ### log complete -c git -n __fish_git_needs_command -a shortlog -d 'Show commit shortlog' complete -c git -n __fish_git_needs_command -a log -d 'Show commit logs' -complete -c git -n '__fish_git_using_command log; and not contains -- -- (commandline -opc)' -a '(__fish_git_ranges)' +complete -c git -n '__fish_git_using_command log; and not contains -- -- (commandline -opc)' -k -a '(__fish_git_ranges)' complete -c git -n '__fish_git_using_command log' -l follow -d 'Continue listing file history beyond renames' complete -c git -n '__fish_git_using_command log' -l no-decorate -d 'Don\'t print ref names' @@ -1555,7 +1555,7 @@ complete -f -c git -n '__fish_git_using_command reset; and not contains -- -- (c ### restore and switch # restore options complete -f -c git -n __fish_git_needs_command -a restore -d 'Restore working tree files' -complete -f -c git -n '__fish_git_using_command restore' -r -s s -l source -d 'Specify the source tree used to restore the working tree' -a '(__fish_git_refs)' +complete -f -c git -n '__fish_git_using_command restore' -r -s s -l source -d 'Specify the source tree used to restore the working tree' -k -a '(__fish_git_refs)' complete -f -c git -n '__fish_git_using_command restore' -s p -l patch -d 'Interactive mode' complete -f -c git -n '__fish_git_using_command restore' -s W -l worktree -d 'Restore working tree (default)' complete -f -c git -n '__fish_git_using_command restore' -s S -l staged -d 'Restore the index' From a24ceaf0df76a475ba470007f1f5a22b64fc6968 Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Thu, 24 Dec 2020 09:42:16 +0100 Subject: [PATCH 015/105] completions/git: offer ranges for cherry-pick --- share/completions/git.fish | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/share/completions/git.fish b/share/completions/git.fish index bceedd3fd..ded4e1f0b 100644 --- a/share/completions/git.fish +++ b/share/completions/git.fish @@ -1016,7 +1016,7 @@ complete -f -c git -n '__fish_git_using_command cherry' -k -a '(__fish_git_refs) ### cherry-pick complete -f -c git -n __fish_git_needs_command -a cherry-pick -d 'Apply the change introduced by an existing commit' -complete -f -c git -n '__fish_git_using_command cherry-pick' -a '(__fish_git_branches --no-merged)' +complete -f -c git -n '__fish_git_using_command cherry-pick' -k -a '(__fish_git_ranges)' # TODO: Filter further complete -f -c git -n '__fish_git_using_command cherry-pick; and __fish_git_possible_commithash' -ka '(__fish_git_commits)' complete -f -c git -n '__fish_git_using_command cherry-pick' -s e -l edit -d 'Edit the commit message prior to committing' From c890982c9037ba9617d61cc614fc9c18664317c0 Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Thu, 24 Dec 2020 10:14:42 +0100 Subject: [PATCH 016/105] GNUMakefile: remove redundant CMake arguments --- GNUmakefile | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/GNUmakefile b/GNUmakefile index 628a23292..ced0ba880 100644 --- a/GNUmakefile +++ b/GNUmakefile @@ -43,9 +43,8 @@ build/fish: build/$(BUILDFILE) # Use build as an order-only dependency. This prevents the target from always being outdated # after a make run, and more importantly, doesn't clobber manually specified CMake options. build/$(BUILDFILE): | build - cd build; $(CMAKE) .. -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -G "$(GENERATOR)" \ - -DCMAKE_INSTALL_PREFIX="$(PREFIX)" -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \ - -DCMAKE_BUILD_TYPE=RelWithDebInfo + cd build; $(CMAKE) .. -G "$(GENERATOR)" \ + -DCMAKE_INSTALL_PREFIX="$(PREFIX)" -DCMAKE_EXPORT_COMPILE_COMMANDS=1 build: mkdir -p build From 69a9785f50a7f520c5d6f7578aa350102ea0bc21 Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Thu, 24 Dec 2020 11:00:20 +0100 Subject: [PATCH 017/105] Refactor: pass by value, not reference, to enable move semantics clang-tidy wrote: > warning: passing result of std::move() as a const reference argument; > no move will actually happen [performance-move-const-arg] --- src/reader.cpp | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/src/reader.cpp b/src/reader.cpp index 62f8c3e1f..cf81c6710 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -643,8 +643,7 @@ class reader_data_t : public std::enable_shared_from_this { /// Erase @length characters starting at @offset. void erase_substring(editable_line_t *el, size_t offset, size_t length); /// Replace the text of length @length at @offset by @replacement. - void replace_substring(editable_line_t *el, size_t offset, size_t length, - const wcstring &replacement); + void replace_substring(editable_line_t *el, size_t offset, size_t length, wcstring replacement); void push_edit(editable_line_t *el, edit_t &&edit); /// Insert the character into the command line buffer and print it to the screen using syntax @@ -1440,8 +1439,8 @@ void reader_data_t::erase_substring(editable_line_t *el, size_t offset, size_t l } void reader_data_t::replace_substring(editable_line_t *el, size_t offset, size_t length, - const wcstring &replacement) { - push_edit(el, edit_t(offset, length, replacement)); + wcstring replacement) { + push_edit(el, edit_t(offset, length, std::move(replacement))); } /// Insert the string in the given command line at the given cursor position. The function checks if From 53d922bde6025ca8505bc7f104ee090eae7c7896 Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Thu, 24 Dec 2020 11:13:39 +0100 Subject: [PATCH 018/105] build_tools/lint.fish: correct cppcheck config location Which was moved in 9b3bfb63d ("cppcheck: Move config files to build_tools") Also get rid of the nonstandard cppcheck output format. --- build_tools/cppcheck.sh | 6 +++--- build_tools/lint.fish | 15 +-------------- 2 files changed, 4 insertions(+), 17 deletions(-) diff --git a/build_tools/cppcheck.sh b/build_tools/cppcheck.sh index 412de8347..178686bb2 100755 --- a/build_tools/cppcheck.sh +++ b/build_tools/cppcheck.sh @@ -1,7 +1,7 @@ #!/bin/sh -cppcheck --enable=all --std=posix --quiet \ - --suppressions-list=build_tools/cppcheck.suppressions \ +cppcheck --std=posix --quiet \ + --suppressions-list=build_tools/cppcheck.suppressions --inline-suppr \ --rule-file=build_tools/cppcheck.rules \ --force \ - ./src/ + ${@:---enable=all ./src/} diff --git a/build_tools/lint.fish b/build_tools/lint.fish index 69e93b3cf..c5b3c824e 100755 --- a/build_tools/lint.fish +++ b/build_tools/lint.fish @@ -83,20 +83,7 @@ if set -q c_files[1] echo ======================================== echo Running cppcheck echo ======================================== - # The stderr to stdout redirection is because cppcheck, incorrectly IMHO, writes its - # diagnostic messages to stderr. Anyone running this who wants to capture its output will - # expect those messages to be written to stdout. - set -l cn (set_color normal) - set -l cb (set_color --bold) - set -l cu (set_color --underline) - set -l cm (set_color magenta) - set -l cbrm (set_color brmagenta) - set -l template "[$cb$cu{file}$cn$cb:{line}$cn] $cbrm{severity}$cm ({id}):$cn\n {message}" - set cppcheck_args -q --verbose --std=c++11 --std=posix --language=c++ --template $template \ - --suppress=missingIncludeSystem --inline-suppr --enable=$cppchecks \ - --rule-file=.cppcheck.rules --suppressions-list=.cppcheck.suppressions $cppcheck_args - - cppcheck $cppcheck_args $c_files 2>&1 + build_tools/cppcheck.sh --enable=$cppchecks $c_files 2>&1 echo echo ======================================== From ad3b76eeb7af8c3de6bea8d4bcd1ffda1ec575dc Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Thu, 24 Dec 2020 11:41:38 +0100 Subject: [PATCH 019/105] CONTRIBUTING: stop recommending deprecated Vim plugin The description on the plugin page says "!!!Deprecated!!!". --- CONTRIBUTING.rst | 5 ----- 1 file changed, 5 deletions(-) diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index c9f10677b..cfbf02b50 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -128,11 +128,6 @@ If you use Vim I recommend the `vim-clang-format plugin `__ by [@rhysd](https://github.com/rhysd). -You can also get Vim to provide reasonably correct behavior by -installing - -http://www.vim.org/scripts/script.php?script_id=2636 - Emacs ^^^^^ From 39a3aa0c2d3efbb7751d82178c3125ec8bcd59df Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Thu, 24 Dec 2020 11:41:38 +0100 Subject: [PATCH 020/105] CONTRIBUTING: shorten and remove stale description We do use "// clang-format off" (once). --- CONTRIBUTING.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index cfbf02b50..811f47c71 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -172,8 +172,8 @@ made to run fish_indent via e.g. Suppressing Reformatting of C++ Code ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If you have a good reason for doing so you can tell ``clang-format`` to -not reformat a block of code by enclosing it in comments like this: +You can tell ``clang-format`` to not reformat a block by enclosing it in +comments like this: :: @@ -181,10 +181,6 @@ not reformat a block of code by enclosing it in comments like this: code to ignore // clang-format on -However, as I write this there are no places in the code where we use -this and I can’t think of any legitimate reasons for exempting blocks of -code from clang-format. - Fish Script Style Guide ----------------------- From f0f5724e18d929e0251624a476c8e83cba1c124d Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Thu, 24 Dec 2020 11:41:38 +0100 Subject: [PATCH 021/105] CONTRIBUTING: Debian provides a "clang-format" package --- CONTRIBUTING.rst | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 811f47c71..2df780982 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -349,7 +349,7 @@ To install the lint checkers on Debian-based Linux distributions: sudo apt-get install oclint sudo apt-get install cppcheck -Installing the Reformatting Tools +Installing the Formatting Tools ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Mac OS X: @@ -362,15 +362,7 @@ Debian-based: :: - apt-cache search clang-format - -Above will list all the versions available. Pick the newest one -available (3.9 for Ubuntu 16.10 as I write this) and install it: - -:: - - sudo apt-get install clang-format-3.9 - sudo ln -s /usr/bin/clang-format-3.9 /usr/bin/clang-format + sudo apt-get install clang-format Message Translations -------------------- From a205225b4e48e464be5adc2545e2cd18c8221958 Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Fri, 25 Dec 2020 09:05:05 +0100 Subject: [PATCH 022/105] lint.fish: properly handle -I and -D args for cppcheck lint.fish receives arguments that contain multiple includes and defines. As a result, we passed arguments like "-I/usr/include -I$HOME/fish-shell/build -I/usr/include" to cppcheck which interprets this as a single include directory. This leads to errors like this one (because the "build" dir was missing): src/common.h:4:0: information: Include file: "config.h" not found. [missingInclude] --- build_tools/lint.fish | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/build_tools/lint.fish b/build_tools/lint.fish index c5b3c824e..5a9e6edb6 100755 --- a/build_tools/lint.fish +++ b/build_tools/lint.fish @@ -18,11 +18,11 @@ argparse a/all p/project= -- $argv # We only want -D and -I options to be passed thru to cppcheck. for arg in $argv if string match -q -- '-D*' $arg - set cppcheck_args $cppcheck_args $arg + set -a cppcheck_args (string split -- ' ' $arg) else if string match -q -- '-I*' $arg - set cppcheck_args $cppcheck_args $arg + set -a cppcheck_args (string split -- ' ' $arg) else if string match -q -- '-iquote*' $arg - set cppcheck_args $cppcheck_args $arg + set -a cppcheck_args (string split -- ' ' $arg) end end From 8fc9b9d61bb39ce7e431d705a93a9a7a3d1fb8d6 Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Fri, 25 Dec 2020 09:25:08 +0100 Subject: [PATCH 023/105] Address some minor lints A mildly interesting one is the call to test_wchar2utf8 with a non-null pointer ("u1"/"dst") but 0 length. In this case we relied on malloc(0) returning non-null which is not guaranteed. src/fish_tests.cpp:1619:23: warning: Call to 'malloc' has an allocation size of 0 bytes [clang-analyzer-optin.portability.UnixAPI] mem = (char *)malloc(dlen); ^ test_wchar2utf8(w1, sizeof(w1) / sizeof(*w1), u1, 0, 0, 0, "invalid params, dst is not NULL"); --- src/fish_tests.cpp | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index d453fcfb9..3d822569d 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -816,7 +816,7 @@ static void test_fd_monitor() { item_maker_t(const item_maker_t &) = delete; // Write 42 bytes to our write end. - void write42() { + void write42() const { char buff[42] = {0}; (void)write_loop(writer.fd(), buff, sizeof buff); } @@ -826,7 +826,7 @@ static void test_fd_monitor() { // Items which will never receive data or be called back. item_maker_t item_never(fd_monitor_item_t::kNoTimeout); - item_maker_t item_hugetimeout(100000000llu * usec_per_msec); + item_maker_t item_hugetimeout(100000000LLU * usec_per_msec); // Item which should get no data, and time out. item_maker_t item0_timeout(16 * usec_per_msec); @@ -1616,7 +1616,8 @@ static void test_wchar2utf8(const wchar_t *src, size_t slen, const char *dst, si #endif if (dst) { - mem = (char *)malloc(dlen); + // We want to pass a valid pointer to wchar_to_utf8, so allocate at least one byte. + mem = (char *)malloc(dlen + 1); if (!mem) { err(L"w2u: %s: MALLOC FAILED", descr); return; @@ -2424,7 +2425,7 @@ struct pager_layout_testcase_t { text.push_back(p.character); } if (text != expected) { - std::fwprintf(stderr, L"width %zu got %zu<%ls>, expected %zu<%ls>\n", this->width, + std::fwprintf(stderr, L"width %d got %zu<%ls>, expected %zu<%ls>\n", this->width, text.length(), text.c_str(), expected.length(), expected.c_str()); for (size_t i = 0; i < std::max(text.length(), expected.length()); i++) { std::fwprintf(stderr, L"i %zu got <%lx> expected <%lx>\n", i, @@ -4107,7 +4108,7 @@ void history_tests_t::test_history_races() { history_t(L"race_test").clear(); pid_t children[RACE_COUNT]; - for (size_t i = 0; i < RACE_COUNT; i++) { + for (pid_t &child : children) { pid_t pid = fork(); if (!pid) { // Child process. @@ -4116,7 +4117,7 @@ void history_tests_t::test_history_races() { exit_without_destructors(0); } else { // Parent process. - children[i] = pid; + child = pid; } } @@ -4374,15 +4375,10 @@ void history_tests_t::test_history_formats() { } else { // The results are in the reverse order that they appear in the bash history file. // We don't expect whitespace to be elided (#4908: except for leading/trailing whitespace) - const wchar_t *expected[] = {L"/** # see issue 7407", - L"sleep 123", - L"a && echo valid construct", - L"final line", - L"echo supsup", - L"export XVAR='exported'", - L"history --help", - L"echo foo", - NULL}; + const wchar_t *expected[] = { + L"/** # see issue 7407", L"sleep 123", L"a && echo valid construct", + L"final line", L"echo supsup", L"export XVAR='exported'", + L"history --help", L"echo foo", NULL}; history_t &test_history = history_t::history_with_name(L"bash_import"); test_history.populate_from_bash(f); if (!history_equals(test_history, expected)) { @@ -4549,6 +4545,10 @@ static bool test_1_parse_ll2(const wcstring &src, wcstring *out_cmd, wcstring *o statement = tmp; } } + if (!statement) { + say(L"No decorated statement found in '%ls'", src.c_str()); + return false; + } // Return its decoration and command. *out_deco = statement->decoration(); From 801955851b9a1bbcc708e762a33790c223e701f0 Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Fri, 25 Dec 2020 13:03:43 +0100 Subject: [PATCH 024/105] Workaround clang-tidy incorrectly assuming null This silences a false positive linter warning about a null dereference. --- src/builtin_complete.cpp | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/builtin_complete.cpp b/src/builtin_complete.cpp index a2d44b348..42e7a0831 100644 --- a/src/builtin_complete.cpp +++ b/src/builtin_complete.cpp @@ -205,6 +205,7 @@ maybe_t builtin_complete(parser_t &parser, io_streams_t &streams, wchar_t * } case 'd': { desc = w.woptarg; + assert(desc); break; } case 'u': { @@ -249,6 +250,7 @@ maybe_t builtin_complete(parser_t &parser, io_streams_t &streams, wchar_t * } case 'a': { comp = w.woptarg; + assert(comp); break; } case 'e': { @@ -257,6 +259,7 @@ maybe_t builtin_complete(parser_t &parser, io_streams_t &streams, wchar_t * } case 'n': { condition = w.woptarg; + assert(condition); break; } case 'w': { From 686d64cf05cd93050c215e21d4c3d0ce0f387e2e Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Fri, 25 Dec 2020 10:04:02 +0100 Subject: [PATCH 025/105] Disable some clang-tidy lints with false positives One warns about using system() which we only use in test code (we're all adults): src/fish_tests.cpp:2015:9: warning: calling 'system' uses a command processor [cert-env33-c] if (system("mkdir -p test/fish_expand_test/bb/")) err(L"mkdir failed"); Some conversion warnings that don't seem very useful: src/input_common.cpp:181:20: warning: 'signed char' to 'wint_t' (aka 'unsigned int') conversion; consider casting to 'unsigned char' first. [cert-str34-c] wint_t b = evt.get_char(); Warning about varargs doesn't make sense, because some of our functions use std::vswprintf() internally. src/ast.cpp:486:10: warning: do not define a C-style variadic function; consider using a function parameter pack or currying instead [cert-dcl50-cpp] void internal_error(const char *func, const wchar_t *fmt, ...) const { Finally, what seems like a false positive; "va" is initialized by va_copy: src/common.cpp:468:18: warning: Function 'vswprintf' is called with an uninitialized va_list argument [clang-analyzer-valist.Uninitialized] status = std::vswprintf(buff, size / sizeof(wchar_t), format, va); --- .clang-tidy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.clang-tidy b/.clang-tidy index 04783fdb4..b463c3d74 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -1,5 +1,5 @@ --- -Checks: 'clang-diagnostic-*,clang-analyzer-*,cert-*,performance-*,portability-*,modernize-use-auto,modernize-loop-convert,modernize-use-bool-literals,modernize-use-using,hicpp-uppercase-literal-suffix,readability-make-member-function-const,readability-redundant-string-init,readability-inconsistent-declaration-parameter-name,readability-redundant-access-specifiers,-performance-noexcept-move-constructor,-cert-dcl37-c,-cert-dcl51-cpp' +Checks: 'clang-diagnostic-*,clang-analyzer-*,-clang-analyzer-valist.Uninitialized,cert-*,performance-*,portability-*,-modernize-use-auto,modernize-loop-convert,modernize-use-bool-literals,modernize-use-using,hicpp-uppercase-literal-suffix,readability-make-member-function-const,readability-redundant-string-init,readability-inconsistent-declaration-parameter-name,readability-redundant-access-specifiers,-performance-noexcept-move-constructor,-cert-dcl37-c,-cert-dcl50-cpp,-cert-dcl51-cpp,-cert-str34-c,-cert-env33-c' WarningsAsErrors: '' HeaderFilterRegex: '' AnalyzeTemporaryDtors: false From f03ff8cd009d82e702650e759e5bf2fc6f03181f Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Wed, 30 Dec 2020 00:40:56 -0800 Subject: [PATCH 026/105] Add a test for history path detection This will support history path detection improvements in a future commit. --- src/fish_tests.cpp | 116 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 103 insertions(+), 13 deletions(-) diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 3d822569d..0a2153a18 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -1884,26 +1884,47 @@ static void test_lru() { do_test(cache.evicted.size() == size_t(total_nodes)); } -/// A crappy environment_t that only knows about PWD. -struct pwd_environment_t : public environment_t { - std::map extras; +/// An environment built around an std::map. +struct test_environment_t : public environment_t { + std::map vars; virtual maybe_t get(const wcstring &key, env_mode_flags_t mode = ENV_DEFAULT) const override { UNUSED(mode); - if (key == L"PWD") { - return env_var_t{wgetcwd(), 0}; + auto iter = vars.find(key); + if (iter != vars.end()) { + return env_var_t(iter->second, ENV_DEFAULT); } - auto extra = extras.find(key); - if (extra != extras.end()) { - return env_var_t(extra->second, ENV_DEFAULT); - } - return {}; + return none(); } wcstring_list_t get_names(int flags) const override { UNUSED(flags); - return {L"PWD"}; + wcstring_list_t result; + for (const auto &kv : vars) { + result.push_back(kv.first); + } + return result; + } +}; + +/// A test environment that knows about PWD. +struct pwd_environment_t : public test_environment_t { + virtual maybe_t get(const wcstring &key, + env_mode_flags_t mode = ENV_DEFAULT) const override { + if (key == L"PWD") { + return env_var_t{wgetcwd(), 0}; + } + return test_environment_t::get(key, mode); + } + + wcstring_list_t get_names(int flags) const override { + auto res = test_environment_t::get_names(flags); + res.clear(); + if (std::count(res.begin(), res.end(), L"PWD") == 0) { + res.push_back(L"PWD"); + } + return res; } }; @@ -3323,7 +3344,7 @@ static void test_autosuggest_suggest_special() { const wcstring wd = L"test/autosuggest_test"; pwd_environment_t vars{}; - vars.extras[L"HOME"] = parser_t::principal_parser().vars().get(L"HOME")->as_string(); + vars.vars[L"HOME"] = parser_t::principal_parser().vars().get(L"HOME")->as_string(); perform_one_autosuggestion_cd_test(L"cd test/autosuggest_test/0", L"foobar/", vars, __LINE__); perform_one_autosuggestion_cd_test(L"cd \"test/autosuggest_test/0", L"foobar/", vars, __LINE__); @@ -3352,7 +3373,7 @@ static void test_autosuggest_suggest_special() { perform_one_autosuggestion_cd_test(L"cd 'test/autosuggest_test/5", L"foo\"bar/", vars, __LINE__); - vars.extras[L"AUTOSUGGEST_TEST_LOC"] = wd; + vars.vars[L"AUTOSUGGEST_TEST_LOC"] = wd; perform_one_autosuggestion_cd_test(L"cd $AUTOSUGGEST_TEST_LOC/0", L"foobar/", vars, __LINE__); perform_one_autosuggestion_cd_test(L"cd ~/test_autosuggest_suggest_specia", L"l/", vars, __LINE__); @@ -3934,6 +3955,7 @@ class history_tests_t { public: static void test_history(); static void test_history_merge(); + static void test_history_path_detection(); static void test_history_formats(); // static void test_history_speed(void); static void test_history_races(); @@ -4286,6 +4308,73 @@ void history_tests_t::test_history_merge() { everything->clear(); } +void history_tests_t::test_history_path_detection() { + // Regression test for #7582. + say(L"Testing history path detection"); + char tmpdirbuff[] = "/tmp/fish_test_history.XXXXXX"; + wcstring tmpdir = str2wcstring(mkdtemp(tmpdirbuff)); + if (! string_suffixes_string(L"/", tmpdir)) { + tmpdir.push_back(L'/'); + } + + // Place one valid file in the directory. + wcstring filename = L"testfile"; + std::string path = wcs2string(tmpdir + filename); + FILE *f = fopen(path.c_str(), "w"); + if (!f) { + err(L"Failed to open test file from history path detection"); + return; + } + fclose(f); + + test_environment_t vars; + vars.vars[L"PWD"] = tmpdir; + vars.vars[L"HOME"] = tmpdir; + + history_t &history = history_t::history_with_name(L"path_detection"); + history.add_pending_with_file_detection(L"cmd0 not/a/valid/path", tmpdir); + history.add_pending_with_file_detection(L"cmd1 " + filename, tmpdir); + history.add_pending_with_file_detection(L"cmd2 " + tmpdir + L"/" + filename, tmpdir); + history.resolve_pending(); + + constexpr size_t hist_size = 3; + if (history.size() != hist_size) { + err(L"history has wrong size: %lu but expected %lu", (unsigned long)history.size(), (unsigned long)hist_size); + history.clear(); + return; + } + + // Expected sets of paths. + wcstring_list_t expected[hist_size] = { + {}, + {filename}, + {tmpdir + L"/" + filename}, + }; + + size_t lap; + const size_t maxlap = 128; + for (lap = 0; lap < maxlap; lap++) { + int failures = 0; + bool last = (lap + 1 == maxlap); + for (size_t i = 1; i <= hist_size; i++) { + if (history.item_at_index(i).required_paths != expected[hist_size - i]) { + failures += 1; + if (last) { + err(L"Wrong detected paths for item %lu", (unsigned long)i); + } + } + } + if (failures == 0) { + break; + } + // The file detection takes a little time since it occurs in the background. + // Loop until the test passes. + usleep(1E6 / 500); // 1 msec + } + //fprintf(stderr, "History saving took %lu laps\n", (unsigned long)lap); + history.clear(); +} + static bool install_sample_history(const wchar_t *name) { wcstring path; if (!path_get_data(path)) { @@ -6198,6 +6287,7 @@ int main(int argc, char **argv) { if (should_test_function("autosuggest_suggest_special")) test_autosuggest_suggest_special(); if (should_test_function("history")) history_tests_t::test_history(); if (should_test_function("history_merge")) history_tests_t::test_history_merge(); + if (should_test_function("history_paths")) history_tests_t::test_history_path_detection(); if (!is_windows_subsystem_for_linux()) { // this test always fails under WSL if (should_test_function("history_races")) history_tests_t::test_history_races(); From bdb99168f017c7e365da2a7285e5c63fb2fe20d6 Mon Sep 17 00:00:00 2001 From: David Adam Date: Thu, 31 Dec 2020 22:06:25 +0800 Subject: [PATCH 027/105] CHANGELOG: work on 3.2.0 --- CHANGELOG.rst | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 050fa6c3b..59f898056 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -11,7 +11,7 @@ Notable improvements and fixes # Show all dmesg lines related to "usb" dmesg -w | string match '*usb*' -- Prompts whose width exceeds $COLUMNS will now be truncated instead of replaced with `"> "` (:issue:`904`). +- Prompts whose width exceeds $COLUMNS will now be truncated instead of replaced with ``"> "`` (:issue:`904`). - When pressing Tab, fish displays ambiguous completions even when they have a common prefix, without the user having to press Tab again (:issue:`6924`). @@ -93,7 +93,7 @@ Scripting improvements - ``set --erase`` and ``abbr --erase`` can now erase multiple things in one go, matching ``functions --erase`` (:issue:`7377`). - ``abbr --erase`` no longer errors on an unset abbreviation (:issue:`7376`). - ``test -t``, for testing whether file descriptors are connected to a terminal, works for file descriptors 0, 1, and 2 (:issue:`4766`). It can still return incorrect results in other cases (:issue:`1228`). -- Trying to run scripts with Windows line endings (CRLF) via the shebang produces a sensible error (:issue:`2783`). +- Trying to execute scripts with Windows line endings (CRLF) produces a sensible error (:issue:`2783`). - An ``alias`` that delegates to a command with the same name no longer triggers an error about recursive completion (:issue:`7389`). - ``math`` now has a ``--base`` option to output the result in hexadecimal or octal (:issue:`7496`) and some more specific errors (:issue:`7508`). - ``math`` learned bitwise functions ``bitand``, ``bitor`` and ``bitxor``, used like ``math "bitand(0xFE, 5)"`` (:issue:`7281`). @@ -129,7 +129,7 @@ Interactive improvements - The output of ``time`` is now properly aligned in all cases (:issue:`6726`). - The ``pwd`` command supports the long options ``--logical`` and ``--physical``, matching other implementations (:issue:`6787`). - The command-not-found handling has been simplified. When it can't find a command, fish now just executes a function called ``fish_command_not_found`` instead of firing an event, making it easier to replace and reason about. Shims for backwards-compatibility have been added (:issue:`7293`). -- Control-C no longer occasionally prints an "unknown command" error (:issue:`7145`). +- Control-C no longer occasionally prints an "unknown command" error (:issue:`7145`) or overwrites multiline prompts (:issue:`3537`). - Autocompletions work properly after Control-C to cancel the commmand line (:issue:`6937`). - History search is now case-insensitive unless the search string contains an uppercase character (:issue:`7273`). - ``fish_update_completions`` has a new ``-keep`` option, which improves speed by skipping completions that already exist (:issue:`6775`). @@ -149,6 +149,7 @@ Interactive improvements - Autosuggestions from history are now case-sensitive, and tab completions are "smartcase": they offer case-insensitive matches if the input string is lowercase (:issue:`3978`). - ``$status`` from completion scripts is no longer visible outside, like in the prompt - this prevents status display in the prompt from being overwritten (:issue:`7555`) - A macOS regarding apropos that was fixed in later 10.15 versions was reintroduced in Big Sur. Fish now works around it again, so command completion isn't super slow anymore (:issue:`7365`). +- Updated localisations for pt_BR (:issue:`#7480`). New or improved bindings ^^^^^^^^^^^^^^^^^^^^^^^^ @@ -175,7 +176,6 @@ New or improved bindings - The definition of "word" and "bigword" for movements was refined, fixing e.g. vi mode's behavior with ``e`` on the second-to-last char, and bigword's behavior with single-char words and non-blank non-graphic characters (:issue:`7353`, :issue:`7354`, :issue:`4025`, :issue:`7328`, :issue:`7325`) - Fish's clipboard bindings now also support WSL via powershell and clip.exe (:issue:`7455`). - Improved prompts ^^^^^^^^^^^^^^^^ @@ -183,7 +183,7 @@ Improved prompts commands prefixed with ``not`` (:issue:`6566`). - git prompts include all untracked files in the repository, not just those in the current directory (:issue:`6086`). -- The git prompts correctly show stash states (:issue:`6876`, :issue:`7136`). +- The git prompts correctly show stash states (:issue:`6876`, :issue:`7136`) and clean states (:issue:`7471`). - The Mercurial prompt correctly shows untracked status (:issue:`6906`). - The ``fish_vcs_prompt`` passes its arguments to the various VCS prompts that it calls (:issue:`7033`). - The Subversion prompt was broken in a number of ways in 3.1.0 and has been restored (:issue:`7278`). @@ -217,7 +217,10 @@ Completions - ``alias`` (:issue:`7035`) - ``apk`` (:issue:`7108`) - ``asciidoctor`` (:issue:`7000`) + - ``bootctl`` (:issue:`7428`) + - ``bluetoothctl`` (:issue:`7438`) - ``cmark`` (:issue:`7000`) + - ``coredumpctl`` (:issue:`7428`) - ``create_ap`` (:issue:`7096`) - ``deno`` (:issue:`7138`) - ``dhclient`` @@ -229,6 +232,9 @@ Completions - ``gh`` (:issue:`7112`) - ``gitk`` - ``hikari`` (:issue:`7083`) + - ``homectl`` (:issue:`7435`) + - ``hostnamectl`` (:issue:`7428`) + - ``icdif`` (:issue:`7503`) - ``imv`` (:issue:`6675`) - ``julia`` (:issue:`7468`) - ``k3d`` (:issue:`7202`) @@ -285,6 +291,7 @@ For distributors and developers - Running the full interactive test suite now requires Python 3.5+ and the pexpect package (:issue:`6825`); the expect package is no longer required. - Support for Python 2 in fish's tools (``fish_config`` and the manual page completion generator) is no longer guaranteed. Please use Python 3.5 or later (:issue:`6537`). - The webconfig tool no longer requires python's distutils (:issue:`7514`) +- fish 3.2 is the last release to support Red Hat Enterprise Linux & CentOS version 7. -------------- From 66c2266ed1b18cf6aafc6acaff217217d945f3bc Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Thu, 31 Dec 2020 16:33:06 -0800 Subject: [PATCH 028/105] Correct a changelog 'issue' template to remove the hash This fixes an 'Invalid issue number' warning. --- CHANGELOG.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 59f898056..d51189a18 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -149,7 +149,7 @@ Interactive improvements - Autosuggestions from history are now case-sensitive, and tab completions are "smartcase": they offer case-insensitive matches if the input string is lowercase (:issue:`3978`). - ``$status`` from completion scripts is no longer visible outside, like in the prompt - this prevents status display in the prompt from being overwritten (:issue:`7555`) - A macOS regarding apropos that was fixed in later 10.15 versions was reintroduced in Big Sur. Fish now works around it again, so command completion isn't super slow anymore (:issue:`7365`). -- Updated localisations for pt_BR (:issue:`#7480`). +- Updated localisations for pt_BR (:issue:`7480`). New or improved bindings ^^^^^^^^^^^^^^^^^^^^^^^^ From 792abf61eca01dd677415db2508de9458f74e460 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Thu, 31 Dec 2020 17:03:53 -0800 Subject: [PATCH 029/105] Attempt to fix the tsan build Deliberately leak the shared thread pool to avoid shutdown dtor registration and tsan complaints at exit. --- src/iothread.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/iothread.cpp b/src/iothread.cpp index ae55049af..57dda3caf 100644 --- a/src/iothread.cpp +++ b/src/iothread.cpp @@ -131,7 +131,8 @@ struct thread_pool_t { /// The thread pool for "iothreads" which are used to lift I/O off of the main thread. /// These are used for completions, etc. -static thread_pool_t s_io_thread_pool(1, IO_MAX_THREADS); +/// Leaked to avoid shutdown dtor registration (including tsan). +static thread_pool_t &s_io_thread_pool = *(new thread_pool_t(1, IO_MAX_THREADS)); static owning_lock> s_result_queue; From 9272703359876bae5ad61bd277e319f77b7ed8fc Mon Sep 17 00:00:00 2001 From: Edouard Lopez Date: Thu, 31 Dec 2020 18:27:37 +0100 Subject: [PATCH 030/105] add ayu colorscheme --- share/tools/web_config/js/colorutils.js | 114 ++++++++++++++++++++++++ 1 file changed, 114 insertions(+) diff --git a/share/tools/web_config/js/colorutils.js b/share/tools/web_config/js/colorutils.js index cedfa9fdf..ecca00a24 100644 --- a/share/tools/web_config/js/colorutils.js +++ b/share/tools/web_config/js/colorutils.js @@ -256,6 +256,120 @@ var color_scheme_fish_default = { }; +var ayuTheme = { + ayu_dark: { + common: { + accent: 'E6B450', + bg: '0A0E14', + fg: 'B3B1AD', + ui: '4D5566' + }, + syntax: { + tag: '39BAE6', + func: 'FFB454', + entity: '59C2FF', + string: 'C2D94C', + regexp: '95E6CB', + markup: 'F07178', + keyword: 'FF8F40', + special: 'E6B673', + comment: '626A73', + constant: 'FFEE99', + operator: 'F29668', + error: 'FF3333' + } + }, + + ayu_light: { + common: { + accent: 'FF9940', + bg: 'FAFAFA', + fg: '575F66', + ui: '8A9199' + }, + syntax: { + tag: '55B4D4', + func: 'F2AE49', + entity: '399EE6', + string: '86B300', + regexp: '4CBF99', + markup: 'F07171', + keyword: 'FA8D3E', + special: 'E6BA7E', + comment: 'ABB0B6', + constant: 'A37ACC', + operator: 'ED9366', + error: 'F51818' + } + }, + + ayu_mirage: { + common: { + accent: 'FFCC66', + bg: '1F2430', + fg: 'CBCCC6', + ui: '707A8C' + }, + syntax: { + tag: '5CCFE6', + func: 'FFD580', + entity: '73D0FF', + string: 'BAE67E', + regexp: '95E6CB', + markup: 'F28779', + keyword: 'FFA759', + special: 'FFE6B3', + comment: '5C6773', + constant: 'D4BFFF', + operator: 'F29E74', + error: 'FF3333' + } + }, + + apply: function(theme, receiver) { + receiver['preferred_background'] = theme.common.bg + receiver['autosuggestion'] = theme.common.ui + receiver['command'] = theme.syntax.tag + receiver['comment'] = theme.syntax.comment + receiver['cwd'] = theme.syntax.entity + receiver['end'] = theme.syntax.operator + receiver['error'] = theme.syntax.error + receiver['escape'] = theme.syntax.regexp + receiver['match'] = theme.syntax.markup + receiver['normal'] = theme.common.fg + receiver['operator'] = theme.syntax.accent + receiver['param'] = theme.common.fg + receiver['quote'] = theme.syntax.string + receiver['redirection'] = theme.syntax.constant + receiver['search_match'] = theme.syntax.accent + receiver['selection'] = theme.syntax.accent + + receiver['colors'] = [] + for (var key in theme) receiver['colors'].push(theme[key]) + }, +} +// ayu Light +var color_scheme_ayu_light = { + name: 'ayu Light', + url: 'https://github.com/edouard-lopez/ayu-theme.fish', +} +ayuTheme.apply(ayuTheme.ayu_light, color_scheme_ayu_light) + +// ayu Dark +var color_scheme_ayu_dark = { + name: 'ayu Dark', + url: 'https://github.com/edouard-lopez/ayu-theme.fish', +} +ayuTheme.apply(ayuTheme.ayu_dark, color_scheme_ayu_dark) + +// ayu Mirage +var color_scheme_ayu_mirage = { + name: 'ayu Mirage', + url: 'https://github.com/edouard-lopez/ayu-theme.fish', +} +ayuTheme.apply(ayuTheme.ayu_mirage, color_scheme_ayu_mirage) + + var TomorrowTheme = { tomorrow_night: {'Background': '1d1f21', 'Current Line': '282a2e', 'Selection': '373b41', 'Foreground': 'c5c8c6', 'Comment': '969896', 'Red': 'cc6666', 'Orange': 'de935f', 'Yellow': 'f0c674', 'Green': 'b5bd68', 'Aqua': '8abeb7', 'Blue': '81a2be', 'Purple': 'b294bb' }, From c8b400bfadbaa515dd3ea252326c314eb2f17a4d Mon Sep 17 00:00:00 2001 From: Edouard Lopez Date: Thu, 31 Dec 2020 18:27:47 +0100 Subject: [PATCH 031/105] register ayu colorscheme --- share/tools/web_config/js/controllers.js | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/share/tools/web_config/js/controllers.js b/share/tools/web_config/js/controllers.js index a09f16479..d58c2428d 100644 --- a/share/tools/web_config/js/controllers.js +++ b/share/tools/web_config/js/controllers.js @@ -79,7 +79,21 @@ controllers.controller("colorsController", function($scope, $http) { $scope.sampleTerminalBackgroundColors = ['white', '#' + solarized.base3, '#300', '#003', '#' + solarized.base03, '#232323', '#'+nord.nord0, 'black']; /* Array of FishColorSchemes */ - $scope.colorSchemes = [color_scheme_fish_default, color_scheme_solarized_light, color_scheme_solarized_dark, color_scheme_tomorrow, color_scheme_tomorrow_night, color_scheme_tomorrow_night_bright, color_scheme_nord, color_scheme_base16_default_dark, color_scheme_base16_default_light, color_scheme_base16_eighties]; + $scope.colorSchemes = [ + color_scheme_fish_default, + color_scheme_ayu_light, + color_scheme_ayu_dark, + color_scheme_ayu_mirage, + color_scheme_solarized_light, + color_scheme_solarized_dark, + color_scheme_tomorrow, + color_scheme_tomorrow_night, + color_scheme_tomorrow_night_bright, + color_scheme_nord, + color_scheme_base16_default_dark, + color_scheme_base16_default_light, + color_scheme_base16_eighties + ]; for (var i=0; i < additional_color_schemes.length; i++) $scope.colorSchemes.push(additional_color_schemes[i]) @@ -301,7 +315,7 @@ controllers.controller("historyController", function($scope, $http, $timeout) { $scope.prevPage = function () { $scope.currentPage = Math.max($scope.currentPage - 1, 0); }; - + $scope.nextPage = function () { $scope.currentPage = Math.min($scope.currentPage + 1, $scope.filteredItemPages.length - 1); From 7ea8e206234f05316b623ac5979153088deb9d9a Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 1 Jan 2021 11:37:25 +0100 Subject: [PATCH 032/105] argparse: Make short flag names optional (#7585) It was always a bit ridiculous that argparse required `X-longflag` if that "X" short flag was never actually used anywhere. Since the short letter is for getopt's benefit, we can hack around this with our old friend: Unicode Private Use Areas. We have a counter, starting at 0xE000 and going to 0xF8FF, that counts up for all options that don't have a short flag and provides one. This gives us up to 6400 long-only options. 6.4K should be enough for everybody. --- doc_src/cmds/argparse.rst | 29 ++++++++++++++----- src/builtin_argparse.cpp | 50 +++++++++++++++++++++++---------- tests/checks/argparse.fish | 57 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 114 insertions(+), 22 deletions(-) diff --git a/doc_src/cmds/argparse.rst b/doc_src/cmds/argparse.rst index eb50145d2..3e05d3faa 100644 --- a/doc_src/cmds/argparse.rst +++ b/doc_src/cmds/argparse.rst @@ -18,7 +18,7 @@ This command makes it easy for fish scripts and functions to handle arguments li Each option specification (``OPTION_SPEC``) is written in the `domain specific language <#option-specifications>`__ described below. All OPTION_SPECs must appear after any argparse flags and before the ``--`` that separates them from the arguments to be parsed. -Each option that is seen in the ARG list will result in a var name of the form ``_flag_X``, where ``X`` is the short flag letter and the long flag name. The OPTION_SPEC always requires a short flag even if it can't be used. So there will always be ``_flag_X`` var set using the short flag letter if the corresponding short or long flag is seen. The long flag name var (e.g., ``_flag_help``) will only be defined, obviously, if the OPTION_SPEC includes a long flag name. +Each option that is seen in the ARG list will result in a var name of the form ``_flag_X``, where ``X`` is the short flag letter and the long flag name. The long flag name var (e.g., ``_flag_help``) will only be defined, obviously, if the OPTION_SPEC includes a long flag name. For example ``_flag_h`` and ``_flag_help`` if ``-h`` or ``--help`` is seen. The var will be set with local scope (i.e., as if the script had done ``set -l _flag_X``). If the flag is a boolean (that is, it just is passed or not, it doesn't have a value) the values are the short and long flags seen. If the option is not a boolean the values will be zero or more values corresponding to the values collected when the ARG list is processed. If the flag was not seen the flag var will not be set. @@ -74,13 +74,11 @@ Option Specifications Each option specification consists of: -- A short flag letter (which is mandatory). It must be an alphanumeric or "#". The "#" character is special and means that a flag of the form ``-123`` is valid. The short flag "#" must be followed by "-" (since the short name isn't otherwise valid since ``_flag_#`` is not a valid var name) and must be followed by a long flag name with no modifiers. +- An optional alphanumeric short flag letter, followed by a ``/`` if the short flag can be used by someone invoking your command or a ``-`` if it should not be exposed as a valid short flag and the letter is just for the ``_flag_X`` variable. -- A ``/`` if the short flag can be used by someone invoking your command else ``-`` if it should not be exposed as a valid short flag. If there is no long flag name these characters should be omitted. You can also specify a '#' to indicate the short and long flag names can be used and the value can be specified as an implicit int; i.e., a flag of the form ``-NNN``. +- An optional long flag name. If not present then only the short flag letter can be used, and if that is not present either it's an error. -- A long flag name which is optional. If not present then only the short flag letter can be used. - -- Nothing if the flag is a boolean that takes no argument or is an implicit int flag, or +- Nothing if the flag is a boolean that takes no argument or is an integer flag, or - ``=`` if it requires a value and only the last instance of the flag is saved, or @@ -94,6 +92,17 @@ See the :ref:`fish_opt ` command for a friendlier but more verbose If a flag is not seen when parsing the arguments then the corresponding _flag_X var(s) will not be set. +Integer flag +------------ + +Sometimes commands take numbers directly as options, like ``foo -55``. To allow this one option spec can have the ``#`` modifier so that any integer will be understood as this flag, and the last number will be given as its value (as if ``=`` was used). + +The ``#`` must follow the short flag letter (if any), and other modifiers like ``=`` are not allowed, except for ``-``:: + + m#maximum + +This does not read numbers given as ``+NNN``, only those that look like flags - ``-NNN``. + Note: Optional arguments ------------------------ @@ -149,6 +158,10 @@ Some OPTION_SPEC examples: - ``h-help`` means that only ``--help`` is valid. The flag is a boolean and can be used more than once. If the long flag is used then ``_flag_h`` and ``_flag_help`` will be set to the count of how many times the long flag was seen. +- ``help`` means that only ``--help`` is valid and only ``_flag_help`` will be set. + +- ``longonly=`` is a flag ``--longonly`` that requires an option, there is no short flag or even short flag variable. + - ``n/name=`` means that both ``-n`` and ``--name`` are valid. It requires a value and can be used at most once. If the flag is seen then ``_flag_n`` and ``_flag_name`` will be set with the single mandatory value associated with the flag. - ``n/name=?`` means that both ``-n`` and ``--name`` are valid. It accepts an optional value and can be used at most once. If the flag is seen then ``_flag_n`` and ``_flag_name`` will be set with the value associated with the flag if one was provided else it will be set with no values. @@ -165,6 +178,8 @@ Some OPTION_SPEC examples: - ``n#max`` means that flags matching the regex "^--?\\d+$" are valid. When seen they are assigned to the variables ``_flag_n`` and ``_flag_max``. This allows any valid positive or negative integer to be specified by prefixing it with a single "-". Many commands support this idiom. For example ``head -3 /a/file`` to emit only the first three lines of /a/file. You can also specify the value using either flag: ``-n NNN`` or ``--max NNN`` in this example. -After parsing the arguments the ``argv`` var is set with local scope to any values not already consumed during flag processing. If there are not unbound values the var is set but ``count $argv`` will be zero. +- ``#longonly`` causes the last integer option to be stored in ``_flag_longonly``. + +After parsing the arguments the ``argv`` var is set with local scope to any values not already consumed during flag processing. If there are no unbound values the var is set but ``count $argv`` will be zero. If an error occurs during argparse processing it will exit with a non-zero status and print error messages to stderr. diff --git a/src/builtin_argparse.cpp b/src/builtin_argparse.cpp index 76f73dc3c..289dbec75 100644 --- a/src/builtin_argparse.cpp +++ b/src/builtin_argparse.cpp @@ -32,7 +32,7 @@ static const wcstring var_name_prefix = L"_flag_"; #define BUILTIN_ERR_INVALID_OPT_SPEC _(L"%ls: Invalid option spec '%ls' at char '%lc'\n") struct option_spec_t { - const wchar_t short_flag; + wchar_t short_flag; wcstring long_flag; wcstring validation_command; wcstring_list_t vals; @@ -208,14 +208,14 @@ static bool parse_flag_modifiers(const argparse_cmd_opts_t &opts, const option_s /// Parse the text following the short flag letter. static bool parse_option_spec_sep(argparse_cmd_opts_t &opts, const option_spec_ref_t &opt_spec, const wcstring &option_spec, const wchar_t **opt_spec_str, - io_streams_t &streams) { + wchar_t &counter, io_streams_t &streams) { const wchar_t *s = *opt_spec_str; if (*(s - 1) == L'#') { if (*s != L'-') { - streams.err.append_format( - _(L"%ls: Short flag '#' must be followed by '-' and a long name\n"), - opts.name.c_str()); - return false; + // Long-only! + s--; + opt_spec->short_flag = counter; + counter++; } if (opts.implicit_int_flag) { streams.err.append_format(_(L"%ls: Implicit int flag '%lc' already defined\n"), @@ -250,9 +250,17 @@ static bool parse_option_spec_sep(argparse_cmd_opts_t &opts, const option_spec_r opt_spec->num_allowed = 1; // mandatory arg and can appear only once s++; // the struct is initialized assuming short_flag_valid should be true } else { - // Long flag name not allowed if second char isn't '/', '-' or '#' so just check for - // behavior modifier chars. - if (!parse_flag_modifiers(opts, opt_spec, option_spec, &s, streams)) return false; + if (*s != L'!' && *s != L'?' && *s != L'=') { + // No short flag separator and no other modifiers, so this is a long only option. + // Since getopt needs a wchar, we have a counter that we count up. + opt_spec->short_flag_valid = false; + s--; + opt_spec->short_flag = counter; + counter++; + } else { + // Try to parse any other flag modifiers + if (!parse_flag_modifiers(opts, opt_spec, option_spec, &s, streams)) return false; + } } *opt_spec_str = s; @@ -261,10 +269,12 @@ static bool parse_option_spec_sep(argparse_cmd_opts_t &opts, const option_spec_r /// This parses an option spec string into a struct option_spec. static bool parse_option_spec(argparse_cmd_opts_t &opts, //!OCLINT(high npath complexity) - const wcstring &option_spec, io_streams_t &streams) { + const wcstring &option_spec, wchar_t &counter, + io_streams_t &streams) { if (option_spec.empty()) { - streams.err.append_format(_(L"%ls: An option spec must have a short flag letter\n"), - opts.name.c_str()); + streams.err.append_format( + _(L"%ls: An option spec must have at least a short or a long flag\n"), + opts.name.c_str()); return false; } @@ -278,7 +288,7 @@ static bool parse_option_spec(argparse_cmd_opts_t &opts, //!OCLINT(high npath c std::unique_ptr opt_spec(new option_spec_t{*s++}); // Try parsing stuff after the short flag. - if (*s && !parse_option_spec_sep(opts, opt_spec, option_spec, &s, streams)) { + if (*s && !parse_option_spec_sep(opts, opt_spec, option_spec, &s, counter, streams)) { return false; } @@ -315,22 +325,32 @@ static int collect_option_specs(argparse_cmd_opts_t &opts, int *optind, int argc io_streams_t &streams) { wchar_t *cmd = argv[0]; + // A counter to give short chars to long-only options because getopt needs that. + // Luckily we have wgetopt so we can use wchars - this is one of the private use areas so we + // have 6400 options available. + wchar_t counter = static_cast(0xE000); + while (true) { if (std::wcscmp(L"--", argv[*optind]) == 0) { ++*optind; break; } - if (!parse_option_spec(opts, argv[*optind], streams)) { + if (!parse_option_spec(opts, argv[*optind], counter, streams)) { return STATUS_CMD_ERROR; } - if (++*optind == argc) { streams.err.append_format(_(L"%ls: Missing -- separator\n"), cmd); return STATUS_INVALID_ARGS; } } + // Check for counter overreach once at the end because this is very unlikely to ever be reached. + if (counter > static_cast(0xF8FF)) { + streams.err.append_format(_(L"%ls: Too many long-only options\n"), cmd); + return STATUS_INVALID_ARGS; + } + if (opts.options.empty()) { streams.err.append_format(_(L"%ls: No option specs were provided\n"), cmd); return STATUS_INVALID_ARGS; diff --git a/tests/checks/argparse.fish b/tests/checks/argparse.fish index 4538727b2..ddeb06ab3 100644 --- a/tests/checks/argparse.fish +++ b/tests/checks/argparse.fish @@ -350,6 +350,54 @@ begin # CHECK: saved_status 57 end +# long-only flags +begin + argparse installed= foo -- --installed=no --foo + set -l + # CHECK: _flag_a 'alpha' 'aaaa' + # CHECK: _flag_b -b + # CHECK: _flag_break -b + # CHECK: _flag_foo --foo + # CHECK: _flag_installed no + # CHECK: _flag_m 1 + # CHECK: _flag_max 1 + # CHECK: argv + # CHECK: saved_status 57 +end + +begin + argparse installed='!_validate_int --max 12' foo -- --installed=5 --foo + set -l + # CHECK: _flag_a 'alpha' 'aaaa' + # CHECK: _flag_b -b + # CHECK: _flag_break -b + # CHECK: _flag_foo --foo + # CHECK: _flag_installed 5 + # CHECK: _flag_m 1 + # CHECK: _flag_max 1 + # CHECK: argv + # CHECK: saved_status 57 +end + +begin + argparse '#num' installed= -- --installed=5 -5 + set -l + # CHECK: _flag_a 'alpha' 'aaaa' + # CHECK: _flag_b -b + # CHECK: _flag_break -b + # CHECK: _flag_installed 5 + # CHECK: _flag_m 1 + # CHECK: _flag_max 1 + # CHECK: _flag_num 5 + # CHECK: argv + # CHECK: saved_status 57 +end + +begin + argparse installed='!_validate_int --max 12' foo -- --foo --installed=error --foo + # CHECKERR: argparse: Value 'error' for flag 'installed' is not an integer +end + # #6483 - error messages for missing arguments argparse -n foo q r/required= -- foo -qr # CHECKERR: foo: Expected argument for option r @@ -440,3 +488,12 @@ function wrongargparse argparse a-b argparse end + +begin + argparse '' + #CHECKERR: argparse: An option spec must have at least a short or a long flag + #CHECKERR: checks/argparse.fish (line {{\d+}}): + #CHECKERR: argparse '' + #CHECKERR: ^ + #CHECKERR: (Type 'help argparse' for related documentation) +end From 8ffa4409369e023756e15f35797661cba117d227 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 1 Jan 2021 11:39:30 +0100 Subject: [PATCH 033/105] More CHANGELOG Changelog, dub dub dub CHANGELOG, dibbie dab dab CHANGELOG --- CHANGELOG.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index d51189a18..8af07a3d5 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -82,7 +82,7 @@ Scripting improvements behaviour (:issue:`7038`). - ``jobs --quiet PID`` no longer prints "no suitable job" if the job for PID does not exist (eg because it has finished) (:issue:`6809`). - All builtins that query if something exists now take ``--query``. ``--quiet`` is deprecated for ``command``, ``jobs`` and ``type`` (:issue:`7276`). -- ``argparse`` now only prints a backtrace with invalid options to argparse itself (:issue:`6703`). +- ``argparse`` no longer requires a short flag letter for long-only options (:issue:`7585`) and only prints a backtrace with invalid options to argparse itself (:issue:`6703`). - ``complete`` takes the first argument as the name of the command if the ``--command``/``-c`` option is not used (``complete git`` is treated like ``complete --command git``), and can show the loaded completions for specific commands with ``complete COMMANDNAME`` (:issue:`7321`). - ``set_color -b`` (without an argument) no longer prints an error message, matching other invalid invocations of this command (:issue:`7154`). - Functions triggered by the ``fish_exit`` event are correctly run when the terminal is closed or the shell receives SIGHUP (:issue:`7014`). @@ -188,6 +188,7 @@ Improved prompts - The ``fish_vcs_prompt`` passes its arguments to the various VCS prompts that it calls (:issue:`7033`). - The Subversion prompt was broken in a number of ways in 3.1.0 and has been restored (:issue:`7278`). - A new helper function ``fish_is_root_user`` simplifies checking for superuser privilege (:issue:`7031`). +- A new pair of colorschemes - ``ayu Light`` and ``ayu Dark`` (:issue:`7596`). Improved terminal support ^^^^^^^^^^^^^^^^^^^^^^^^^ From 5c09a6d91e7d85470852a1991c45533f3d384c76 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 1 Jan 2021 12:19:12 +0100 Subject: [PATCH 034/105] CHANGELOG: Missed one ayu colorscheme *Ozzy voice* I'm going through CHANGELOGs --- CHANGELOG.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 8af07a3d5..06cf6aed4 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -188,7 +188,7 @@ Improved prompts - The ``fish_vcs_prompt`` passes its arguments to the various VCS prompts that it calls (:issue:`7033`). - The Subversion prompt was broken in a number of ways in 3.1.0 and has been restored (:issue:`7278`). - A new helper function ``fish_is_root_user`` simplifies checking for superuser privilege (:issue:`7031`). -- A new pair of colorschemes - ``ayu Light`` and ``ayu Dark`` (:issue:`7596`). +- New colorschemes - ``ayu Light``, ``ayu Dark`` and ``ayu Mirage`` (:issue:`7596`). Improved terminal support ^^^^^^^^^^^^^^^^^^^^^^^^^ From 9231956f62c5061d0811f02beb4382e1e3a3b8f2 Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Fri, 1 Jan 2021 12:15:43 +0100 Subject: [PATCH 035/105] CHANGELOG: document some changes with no associated issue This should cover my remaining user-facing commits since 3.1.2. --- CHANGELOG.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 06cf6aed4..ac4fb88c7 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -63,8 +63,9 @@ Scripting improvements - Computed ("electric") variables such as ``status`` are now only global in scope, so ``set -Uq status`` returns false (:issue:`7032`). - The output for ``set --show`` has been shortened, only mentioning the scopes in which a variable exists (:issue:`6944`). - A new ``fish_posterror`` event is emitted when attempting to execute a command with syntax errors (:issue:`6880`). -- ``fish_indent`` now removes spurious quotes in simple cases (:issue:`6722`) +- ``fish_indent`` now removes unnecessary quotes in simple cases (:issue:`6722`) and learned a ``--check`` option to just check if a file is indented correctly (:issue:`7251`). +- ``fish_indent`` indents continuation lines that follow a line ending in a backslash, ``|``, ``&&`` or ``||``. - ``pushd`` only adds a directory to the stack if changing to it was successful (:issue:`6947`). - A new ``fish_job_summary`` function is called whenever a background job stops or ends, or any job terminates from a signal (:issue:`6959`). @@ -160,6 +161,7 @@ New or improved bindings - Vi mode bindings now support ``dh``, ``dl``, ``c0``, ``cf``, ``ct``, ``cF``, ``cT``, ``ch``, ``cl``, ``y0``, ``ci``, ``ca``, ``yi``, ``ya``, ``di``, ``da``, ``o``, ``O`` and Control+left/right keys to navigate by word (:issue:`6648`, :issue:`6755`, :issue:`6769`, :issue:`7442`). - Vi mode bindings support ``~`` (tilde) to toggle the case of the selected character (:issue:`6908`). - Functions ``up-or-search`` and ``down-or-search`` (up-arrow and down-arrow) can cross empty lines and don't activate search mode if the search fails which makes it easier to use them to move between lines in some situations. +- If history search fails to find a match, the cursor is no longer moved. This is useful when accidentally starting a history search on a multi-line commandline. - The readline command ``beginning-of-history`` (Page Up) now moves to the oldest search instead of the youngest - that's ``end-of-history`` (Page Down). - A new readline command ``forward-single-char`` moves one character to the right, and if an autosuggestion is available, only take a single character from it (:issue:`7217`). - Readline commands can now be joined with ``or`` as a modifier (adding to ``and``), though only some commands report success or failure (:issue:`7217`). @@ -267,7 +269,9 @@ Completions - Lots of improvements to completions. - Improvements to the manpage completion generator (:issue:`7086`). - Significant performance improvements to completion of the available commands (:issue:`7153`). +- ``__fish_complete_suffix`` now uses the same fuzzy matching logic as normal file completion. - ``__fish_complete_suffix`` completes any file but sorts files with matching suffix first (:issue:`7040`). Previously, it only completed files with matching suffix. +- Completions for ``git`` learned to complete the right and left parts of a commit range like ``from..to`` or ``left...right``. Deprecations and removed features --------------------------------- From 57d23c390bd6d56203e54e41e84d957fcce131d4 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 1 Jan 2021 13:58:41 +0100 Subject: [PATCH 036/105] docs: Reword argparse a bit In particular use "variable" instead of "var". --- doc_src/cmds/argparse.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc_src/cmds/argparse.rst b/doc_src/cmds/argparse.rst index 3e05d3faa..d9b42dd09 100644 --- a/doc_src/cmds/argparse.rst +++ b/doc_src/cmds/argparse.rst @@ -18,9 +18,9 @@ This command makes it easy for fish scripts and functions to handle arguments li Each option specification (``OPTION_SPEC``) is written in the `domain specific language <#option-specifications>`__ described below. All OPTION_SPECs must appear after any argparse flags and before the ``--`` that separates them from the arguments to be parsed. -Each option that is seen in the ARG list will result in a var name of the form ``_flag_X``, where ``X`` is the short flag letter and the long flag name. The long flag name var (e.g., ``_flag_help``) will only be defined, obviously, if the OPTION_SPEC includes a long flag name. +Each option that is seen in the ARG list will result in variables named ``_flag_X``, where ``X`` is the short flag letter and the long flag name (if they are defined). For example a ``--help`` option could cause argparse to define one variable called ``_flag_h`` and another called ``_flag_help``. -For example ``_flag_h`` and ``_flag_help`` if ``-h`` or ``--help`` is seen. The var will be set with local scope (i.e., as if the script had done ``set -l _flag_X``). If the flag is a boolean (that is, it just is passed or not, it doesn't have a value) the values are the short and long flags seen. If the option is not a boolean the values will be zero or more values corresponding to the values collected when the ARG list is processed. If the flag was not seen the flag var will not be set. +The variables will be set with local scope (i.e., as if the script had done ``set -l _flag_X``). If the flag is a boolean (that is, it just is passed or not, it doesn't have a value) the values are the short and long flags seen. If the option is not a boolean the values will be zero or more values corresponding to the values collected when the ARG list is processed. If the flag was not seen the flag variable will not be set. Options ------- @@ -180,6 +180,6 @@ Some OPTION_SPEC examples: - ``#longonly`` causes the last integer option to be stored in ``_flag_longonly``. -After parsing the arguments the ``argv`` var is set with local scope to any values not already consumed during flag processing. If there are no unbound values the var is set but ``count $argv`` will be zero. +After parsing the arguments the ``argv`` variable is set with local scope to any values not already consumed during flag processing. If there are no unbound values the variable is set but ``count $argv`` will be zero. If an error occurs during argparse processing it will exit with a non-zero status and print error messages to stderr. From 164a5ebe8137e27ba1989544e3c4e2b9eaa336a4 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 1 Jan 2021 14:18:17 +0100 Subject: [PATCH 037/105] tests: Remove unused colordiff function --- tests/test_util.fish | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/tests/test_util.fish b/tests/test_util.fish index 950a518f0..12811c63b 100644 --- a/tests/test_util.fish +++ b/tests/test_util.fish @@ -125,21 +125,6 @@ function say -V suppress_color end end -function colordiff -d 'Colored diff output for unified diffs' - diff $argv | while read -l line - switch $line - case '+*' - say green $line - case '-*' - say red $line - case '@*' - say cyan $line - case '*' - echo $line - end - end -end - # lame timer for program in {g,}date if command -q $program && $program --version 1>/dev/null 2>/dev/null From b43a8688fe787dba66cbfb086594cdd6f311ed52 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 1 Jan 2021 14:21:20 +0100 Subject: [PATCH 038/105] docs: Correct argparse on short- options These aren't exposed as variables at all, so it's just entirely vestigial now and only kept for backwards compatibility. --- doc_src/cmds/argparse.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc_src/cmds/argparse.rst b/doc_src/cmds/argparse.rst index d9b42dd09..5c44ce22c 100644 --- a/doc_src/cmds/argparse.rst +++ b/doc_src/cmds/argparse.rst @@ -74,7 +74,7 @@ Option Specifications Each option specification consists of: -- An optional alphanumeric short flag letter, followed by a ``/`` if the short flag can be used by someone invoking your command or a ``-`` if it should not be exposed as a valid short flag and the letter is just for the ``_flag_X`` variable. +- An optional alphanumeric short flag letter, followed by a ``/`` if the short flag can be used by someone invoking your command or, for backwards compatibility, a ``-`` if it should not be exposed as a valid short flag (in which case it will also not be exposed as a flag variable). - An optional long flag name. If not present then only the short flag letter can be used, and if that is not present either it's an error. @@ -97,7 +97,7 @@ Integer flag Sometimes commands take numbers directly as options, like ``foo -55``. To allow this one option spec can have the ``#`` modifier so that any integer will be understood as this flag, and the last number will be given as its value (as if ``=`` was used). -The ``#`` must follow the short flag letter (if any), and other modifiers like ``=`` are not allowed, except for ``-``:: +The ``#`` must follow the short flag letter (if any), and other modifiers like ``=`` are not allowed, except for ``-`` (for backwards compatibility):: m#maximum From 364692fa3d935ff31714dd3b8d9d791b32d91ea3 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 1 Jan 2021 18:38:20 +0100 Subject: [PATCH 039/105] CHANGELOG: MOAR --- CHANGELOG.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index ac4fb88c7..7bb3e18f6 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -148,9 +148,10 @@ Interactive improvements - Resuming a piped job by its number, like ``fg %1`` has been fixed (:issue:`7406`). - Commands run from key bindings now use the same tty modes as normal commands (:issue:`7483`). - Autosuggestions from history are now case-sensitive, and tab completions are "smartcase": they offer case-insensitive matches if the input string is lowercase (:issue:`3978`). -- ``$status`` from completion scripts is no longer visible outside, like in the prompt - this prevents status display in the prompt from being overwritten (:issue:`7555`) +- ``$status`` from completion scripts is no longer visible outside, like in the prompt - this prevents status display in the prompt from being overwritten (:issue:`7555`). - A macOS regarding apropos that was fixed in later 10.15 versions was reintroduced in Big Sur. Fish now works around it again, so command completion isn't super slow anymore (:issue:`7365`). - Updated localisations for pt_BR (:issue:`7480`). +- ``fish_trace`` output now starts with ``->`` like ``fish --profile``'s, making the depth more visible (:issue:`7538`). New or improved bindings ^^^^^^^^^^^^^^^^^^^^^^^^ @@ -272,6 +273,9 @@ Completions - ``__fish_complete_suffix`` now uses the same fuzzy matching logic as normal file completion. - ``__fish_complete_suffix`` completes any file but sorts files with matching suffix first (:issue:`7040`). Previously, it only completed files with matching suffix. - Completions for ``git`` learned to complete the right and left parts of a commit range like ``from..to`` or ``left...right``. +- The ``__fish_print_packages`` function was broken apart into one function per package manager, and any completion now only calls its specific function. This helps if multiple package managers are installed on a system (e.g. to create containers). ``__fish_print_packages`` remains as a stub that calls all functions (:issue:`7542`). +- Many completions have their descriptions shortened to fit more options on the screen (:issue:`6981`, :issue:`7550`, :issue:`7109`, :issue:`7569`, :issue:`7081`, :issue:`7291`, :issue:`7163`, :issue:`7378`). +- The ``make`` completions no longer second-guess make's file detection, fixing target completion in some cases (:issue:`7535`) Deprecations and removed features --------------------------------- From 8a585bb711e176435cce07503c37d91410c1df75 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 1 Jan 2021 20:21:45 +0100 Subject: [PATCH 040/105] Also disable winch handling in alacritty It also reflows. We might want to think about doing something more extensible here, as konsole is also about to add reflow, but for now the main problem children here are VTE and alacritty. Extends #7491. --- CHANGELOG.rst | 2 +- share/functions/__fish_config_interactive.fish | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 7bb3e18f6..0700893be 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -209,7 +209,7 @@ Improved terminal support - An issue producing strange status output from commands involving ``not`` has been fixed (:issue:`6566`). - Long command lines are wrapped in all cases, instead of sometimes being put on a new line (:issue:`5118`). - The pager is properly rendered with long command lines selected (:issue:`2557`). -- Fish no longer performs its own resizing in VTE-based terminals, as they perform their own reflowing, which clashes especially with right prompts (:issue:`7491`). +- Fish no longer performs its own resizing in VTE-based terminals and alacritty, as they perform their own reflowing, which clashes especially with right prompts (:issue:`7491`). - Fish now sets terminal modes sooner, which stops output from appearing before the greeting and prompt are ready (:issue:`7489`). Completions diff --git a/share/functions/__fish_config_interactive.fish b/share/functions/__fish_config_interactive.fish index 68209dd2b..35c557f25 100644 --- a/share/functions/__fish_config_interactive.fish +++ b/share/functions/__fish_config_interactive.fish @@ -247,6 +247,10 @@ function __fish_config_interactive -d "Initializations that should be performed if set -q VTE_VERSION return end + # Same for alacritty + if string match -q -- 'alacritty*' $TERM + return + end commandline -f repaint >/dev/null 2>/dev/null end From eb43fc83c5e9e73a9a9f19fbd8cdbd491391a934 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 1 Jan 2021 20:47:23 +0100 Subject: [PATCH 041/105] CHANGELOG: Add that numbered debugging is no more --- CHANGELOG.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 0700893be..8749fde30 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -118,7 +118,7 @@ Interactive improvements - A new variable ``$status_generation`` is incremented only when the previous command produces a status (:issue:`6815`). This can be used, for example, to check whether a failure status is a holdover due to a background job, or actually produced by the last run command. - ``fish_greeting`` is now a function that reads a variable of the same name, and defaults to setting it globally. This removes a universal variable by default and helps with updating the greeting. However, to disable the greeting it is now necessary to explicitly specify universal scope (``set -U fish_greeting``) or to disable it in config.fish (:issue:`7265`). - Events are properly emitted after a job is cancelled (:issue:`2356`). -- A number of new debugging categories have been added, including ``config``, ``path``, ``reader`` and ``screen`` (:issue:`6511`). See the output of ``fish --print-debug-categories`` for the full list. +- A number of new debugging categories have been added, including ``config``, ``path``, ``reader`` and ``screen`` (:issue:`6511`). See the output of ``fish --print-debug-categories`` for the full list. The old numbered debugging levels have been removed. - The enabled debug categories are now printed on shell startup (:issue:`7007`). - The ``-o`` short option to fish, for ``--debug-output``, works correctly instead of producing an invalid option error (:issue:`7254`). From cf8219e3ce8bd2c7f3a2337328806fa21b6fafb1 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 1 Jan 2021 21:15:09 +0100 Subject: [PATCH 042/105] Exit if --no-execute is enabled don't interactively read from the terminal Don't go into implicit interactive mode without ever executing anything - not even `exit` or reacting to ctrl-d. That just renders the shell useless and unquittable. --- src/fish.cpp | 4 ++++ tests/checks/invocation.fish | 3 +++ tests/pexpects/generic.py | 3 +++ 3 files changed, 10 insertions(+) diff --git a/src/fish.cpp b/src/fish.cpp index c04167661..f5e198d14 100644 --- a/src/fish.cpp +++ b/src/fish.cpp @@ -503,6 +503,10 @@ int main(int argc, char **argv) { parser.libdata().exit_current_script = false; } else if (my_optind == argc) { // Implicitly interactive mode. + if (opts.no_exec && isatty(STDIN_FILENO)) { + FLOGF(error, L"no-execute mode enabled and no script given. Exiting"); + return EXIT_FAILURE; // above line should always exit + } res = reader_read(parser, STDIN_FILENO, {}); } else { const char *file = *(argv + (my_optind++)); diff --git a/tests/checks/invocation.fish b/tests/checks/invocation.fish index e258351df..005d3e501 100644 --- a/tests/checks/invocation.fish +++ b/tests/checks/invocation.fish @@ -53,3 +53,6 @@ $fish -c 'string escape y$argv' -c 'string escape x$argv' 1 2 3 # CHECK: x1 # CHECK: x2 # CHECK: x3 + +# Should just do nothing. +$fish --no-execute diff --git a/tests/pexpects/generic.py b/tests/pexpects/generic.py index 030e2e230..efff0bcb5 100644 --- a/tests/pexpects/generic.py +++ b/tests/pexpects/generic.py @@ -51,5 +51,8 @@ expect_prompt("hoge") sendline("echo hoge >| \n cat") expect_prompt("hoge") +sendline("$fish --no-execute 2>&1") +expect_prompt("error: no-execute mode enabled and no script given. Exiting") + sendline("source; or echo failed") expect_prompt("failed") From f547c2fda8671953715a6f272ca7a3dcafd0b07a Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sat, 2 Jan 2021 16:20:54 +0100 Subject: [PATCH 043/105] Webconfig: Fix customizing ayu themes These used a different object format, so they were passed to interpret_color wrong. Because the "common" and "syntax" division doesn't really help all that much, let's just flatten the thing. See #7596. --- share/tools/web_config/js/colorutils.js | 140 +++++++++++------------- 1 file changed, 64 insertions(+), 76 deletions(-) diff --git a/share/tools/web_config/js/colorutils.js b/share/tools/web_config/js/colorutils.js index ecca00a24..ddab52206 100644 --- a/share/tools/web_config/js/colorutils.js +++ b/share/tools/web_config/js/colorutils.js @@ -258,91 +258,79 @@ var color_scheme_fish_default = { var ayuTheme = { ayu_dark: { - common: { - accent: 'E6B450', - bg: '0A0E14', - fg: 'B3B1AD', - ui: '4D5566' - }, - syntax: { - tag: '39BAE6', - func: 'FFB454', - entity: '59C2FF', - string: 'C2D94C', - regexp: '95E6CB', - markup: 'F07178', - keyword: 'FF8F40', - special: 'E6B673', - comment: '626A73', - constant: 'FFEE99', - operator: 'F29668', - error: 'FF3333' - } + 'accent': 'E6B450', + 'bg': '0A0E14', + 'fg': 'B3B1AD', + 'ui': '4D5566', + 'tag': '39BAE6', + 'func': 'FFB454', + 'entity': '59C2FF', + 'string': 'C2D94C', + 'regexp': '95E6CB', + 'markup': 'F07178', + 'keyword': 'FF8F40', + 'special': 'E6B673', + 'comment': '626A73', + 'constant': 'FFEE99', + 'operator': 'F29668', + 'error': 'FF3333', }, ayu_light: { - common: { - accent: 'FF9940', - bg: 'FAFAFA', - fg: '575F66', - ui: '8A9199' - }, - syntax: { - tag: '55B4D4', - func: 'F2AE49', - entity: '399EE6', - string: '86B300', - regexp: '4CBF99', - markup: 'F07171', - keyword: 'FA8D3E', - special: 'E6BA7E', - comment: 'ABB0B6', - constant: 'A37ACC', - operator: 'ED9366', - error: 'F51818' - } + 'accent': 'FF9940', + 'bg': 'FAFAFA', + 'fg': '575F66', + 'ui': '8A9199', + 'tag': '55B4D4', + 'func': 'F2AE49', + 'entity': '399EE6', + 'string': '86B300', + 'regexp': '4CBF99', + 'markup': 'F07171', + 'keyword': 'FA8D3E', + 'special': 'E6BA7E', + 'comment': 'ABB0B6', + 'constant': 'A37ACC', + 'operator': 'ED9366', + 'error': 'F51818', }, ayu_mirage: { - common: { - accent: 'FFCC66', - bg: '1F2430', - fg: 'CBCCC6', - ui: '707A8C' - }, - syntax: { - tag: '5CCFE6', - func: 'FFD580', - entity: '73D0FF', - string: 'BAE67E', - regexp: '95E6CB', - markup: 'F28779', - keyword: 'FFA759', - special: 'FFE6B3', - comment: '5C6773', - constant: 'D4BFFF', - operator: 'F29E74', - error: 'FF3333' - } + 'accent': 'FFCC66', + 'bg': '1F2430', + 'fg': 'CBCCC6', + 'ui': '707A8C', + 'tag': '5CCFE6', + 'func': 'FFD580', + 'entity': '73D0FF', + 'string': 'BAE67E', + 'regexp': '95E6CB', + 'markup': 'F28779', + 'keyword': 'FFA759', + 'special': 'FFE6B3', + 'comment': '5C6773', + 'constant': 'D4BFFF', + 'operator': 'F29E74', + 'error': 'FF3333', }, apply: function(theme, receiver) { - receiver['preferred_background'] = theme.common.bg - receiver['autosuggestion'] = theme.common.ui - receiver['command'] = theme.syntax.tag - receiver['comment'] = theme.syntax.comment - receiver['cwd'] = theme.syntax.entity - receiver['end'] = theme.syntax.operator - receiver['error'] = theme.syntax.error - receiver['escape'] = theme.syntax.regexp - receiver['match'] = theme.syntax.markup - receiver['normal'] = theme.common.fg - receiver['operator'] = theme.syntax.accent - receiver['param'] = theme.common.fg - receiver['quote'] = theme.syntax.string - receiver['redirection'] = theme.syntax.constant - receiver['search_match'] = theme.syntax.accent - receiver['selection'] = theme.syntax.accent + receiver['preferred_background'] = theme.bg + receiver['autosuggestion'] = theme.ui + receiver['command'] = theme.tag + receiver['comment'] = theme.comment + receiver['cwd'] = theme.entity + receiver['end'] = theme.operator + receiver['error'] = theme.error + receiver['escape'] = theme.regexp + receiver['match'] = theme.markup + receiver['normal'] = theme.fg + receiver['operator'] = theme.accent + receiver['param'] = theme.fg + receiver['quote'] = theme.string + receiver['redirection'] = theme.constant + receiver['search_match'] = theme.accent + receiver['selection'] = theme.accent receiver['colors'] = [] for (var key in theme) receiver['colors'].push(theme[key]) From ab5608ddf2990b39c718705f81310daa7798d2d3 Mon Sep 17 00:00:00 2001 From: David Adam Date: Sat, 2 Jan 2021 23:26:40 +0800 Subject: [PATCH 044/105] CHANGELOG: work on 3.2.0 --- CHANGELOG.rst | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 8749fde30..b056cff82 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -139,19 +139,20 @@ Interactive improvements - Long command lines no longer add a blank line after execution (:issue:`6826`) and behave better with backspace (:issue:`6951`). - ``functions -t`` works like the long option ``--handlers-type``, as documented, instead of producing an error (:issue:`6985`). - History search now flashes when it found no more results (:issue:`7362`) -- Fish's debugging can now also be enabled via $FISH_DEBUG and $FISH_DEBUG_OUTPUT from the outside. This helps with debugging when no commandline options can be passed, like when fish is called in a shebang (:issue:`7359`). -- Fish now creates $XDG_RUNTIME_DIR if it does not exist (:issue:`7335`). +- fish's debugging can now also be enabled via $FISH_DEBUG and $FISH_DEBUG_OUTPUT from the outside. This helps with debugging when no commandline options can be passed, like when fish is called in a shebang (:issue:`7359`). +- fish now creates $XDG_RUNTIME_DIR if it does not exist (:issue:`7335`). - ``set_color --print-colors`` now also respects the bold, dim, underline, reverse, italic and background modifiers, to better show their effect (:issue:`7314`). - The fish Web configuration tool (``fish_config``) shows prompts correctly on Termux for Android (:issue:`7298`) and detects Windows Services for Linux 2 properly (:issue:`7027`). - ``funcsave`` has a new ``--directory`` option to specify the location of the saved function (:issue:`7041`). - ``help`` works properly on MSYS2 (:issue:`7113`). -- Resuming a piped job by its number, like ``fg %1`` has been fixed (:issue:`7406`). -- Commands run from key bindings now use the same tty modes as normal commands (:issue:`7483`). +- Resuming a piped job by its number, like ``fg %1``, works correctly (:issue:`7406`). Resumed jobs show the correct title in the terminal emulator (:issue:`7444`). +- Commands run from key bindings now use the same TTY modes as normal commands (:issue:`7483`). - Autosuggestions from history are now case-sensitive, and tab completions are "smartcase": they offer case-insensitive matches if the input string is lowercase (:issue:`3978`). - ``$status`` from completion scripts is no longer visible outside, like in the prompt - this prevents status display in the prompt from being overwritten (:issue:`7555`). -- A macOS regarding apropos that was fixed in later 10.15 versions was reintroduced in Big Sur. Fish now works around it again, so command completion isn't super slow anymore (:issue:`7365`). - Updated localisations for pt_BR (:issue:`7480`). - ``fish_trace`` output now starts with ``->`` like ``fish --profile``'s, making the depth more visible (:issue:`7538`). +- Resizing the terminal window no longer produces a corrupted prompt (:issue:`6532`). +- ``functions`` produces an error rather than crashing on certain invalid arguments (:issue:`7515`). New or improved bindings ^^^^^^^^^^^^^^^^^^^^^^^^ @@ -159,7 +160,7 @@ New or improved bindings - As mentioned above, new readline commands ``undo`` (Control+\_ or Control+Z) and ``redo`` (Alt-/) can be used to revert changes to the command line or the pager search field (:issue:`6570`). - Control-Z is now available for binding (:issue:`7152`). - Additionally, using the ``cancel`` readline command (bound to escape by default) right after fish picked an unambiguous completion will undo that (:issue:`7433`). -- Vi mode bindings now support ``dh``, ``dl``, ``c0``, ``cf``, ``ct``, ``cF``, ``cT``, ``ch``, ``cl``, ``y0``, ``ci``, ``ca``, ``yi``, ``ya``, ``di``, ``da``, ``o``, ``O`` and Control+left/right keys to navigate by word (:issue:`6648`, :issue:`6755`, :issue:`6769`, :issue:`7442`). +- Vi mode bindings now support ``dh``, ``dl``, ``c0``, ``cf``, ``ct``, ``cF``, ``cT``, ``ch``, ``cl``, ``y0``, ``ci``, ``ca``, ``yi``, ``ya``, ``di``, ``da``, ``d;``, ``d,``, ``o``, ``O`` and Control+left/right keys to navigate by word (:issue:`6648`, :issue:`6755`, :issue:`6769`, :issue:`7442`, :issue:`7516`). - Vi mode bindings support ``~`` (tilde) to toggle the case of the selected character (:issue:`6908`). - Functions ``up-or-search`` and ``down-or-search`` (up-arrow and down-arrow) can cross empty lines and don't activate search mode if the search fails which makes it easier to use them to move between lines in some situations. - If history search fails to find a match, the cursor is no longer moved. This is useful when accidentally starting a history search on a multi-line commandline. @@ -177,7 +178,7 @@ New or improved bindings from history if the commandline is empty (:issue:`7137`). - ``__fish_whatis_current_token`` (Alt-W) prints descriptions for functions and builtins (:issue:`7191`). - The definition of "word" and "bigword" for movements was refined, fixing e.g. vi mode's behavior with ``e`` on the second-to-last char, and bigword's behavior with single-char words and non-blank non-graphic characters (:issue:`7353`, :issue:`7354`, :issue:`4025`, :issue:`7328`, :issue:`7325`) -- Fish's clipboard bindings now also support WSL via powershell and clip.exe (:issue:`7455`). +- fish's clipboard bindings now also support WSL via powershell and clip.exe (:issue:`7455`). Improved prompts ^^^^^^^^^^^^^^^^ @@ -189,7 +190,7 @@ Improved prompts - The git prompts correctly show stash states (:issue:`6876`, :issue:`7136`) and clean states (:issue:`7471`). - The Mercurial prompt correctly shows untracked status (:issue:`6906`). - The ``fish_vcs_prompt`` passes its arguments to the various VCS prompts that it calls (:issue:`7033`). -- The Subversion prompt was broken in a number of ways in 3.1.0 and has been restored (:issue:`7278`). +- The Subversion prompt was broken in a number of ways in 3.1.0 and has been restored (:issue:`6715`, :issue:`7278`). - A new helper function ``fish_is_root_user`` simplifies checking for superuser privilege (:issue:`7031`). - New colorschemes - ``ayu Light``, ``ayu Dark`` and ``ayu Mirage`` (:issue:`7596`). @@ -209,8 +210,8 @@ Improved terminal support - An issue producing strange status output from commands involving ``not`` has been fixed (:issue:`6566`). - Long command lines are wrapped in all cases, instead of sometimes being put on a new line (:issue:`5118`). - The pager is properly rendered with long command lines selected (:issue:`2557`). -- Fish no longer performs its own resizing in VTE-based terminals and alacritty, as they perform their own reflowing, which clashes especially with right prompts (:issue:`7491`). -- Fish now sets terminal modes sooner, which stops output from appearing before the greeting and prompt are ready (:issue:`7489`). +- Sessions with right prompts can be resized correctly in GNOME Terminal (and other VTE-based terminals) and Alacritty (:issue:`7491`). +- fish now sets terminal modes sooner, which stops output from appearing before the greeting and prompt are ready (:issue:`7489`). Completions ^^^^^^^^^^^ @@ -238,7 +239,7 @@ Completions - ``hikari`` (:issue:`7083`) - ``homectl`` (:issue:`7435`) - ``hostnamectl`` (:issue:`7428`) - - ``icdif`` (:issue:`7503`) + - ``icdiff`` (:issue:`7503`) - ``imv`` (:issue:`6675`) - ``julia`` (:issue:`7468`) - ``k3d`` (:issue:`7202`) @@ -269,7 +270,7 @@ Completions - Lots of improvements to completions. - Improvements to the manpage completion generator (:issue:`7086`). -- Significant performance improvements to completion of the available commands (:issue:`7153`). +- Significant performance improvements to completion of the available commands (:issue:`7153`), especially on macOS Big Sur where there was a significant regression (:issue:`7365`). - ``__fish_complete_suffix`` now uses the same fuzzy matching logic as normal file completion. - ``__fish_complete_suffix`` completes any file but sorts files with matching suffix first (:issue:`7040`). Previously, it only completed files with matching suffix. - Completions for ``git`` learned to complete the right and left parts of a commit range like ``from..to`` or ``left...right``. @@ -299,7 +300,7 @@ For distributors and developers codesigning is enabled (:issue:`6952`). - Running the full interactive test suite now requires Python 3.5+ and the pexpect package (:issue:`6825`); the expect package is no longer required. - Support for Python 2 in fish's tools (``fish_config`` and the manual page completion generator) is no longer guaranteed. Please use Python 3.5 or later (:issue:`6537`). -- The webconfig tool no longer requires python's distutils (:issue:`7514`) +- The web-based configuration tool no longer requires Python's distutils (:issue:`7514`) - fish 3.2 is the last release to support Red Hat Enterprise Linux & CentOS version 7. -------------- From cdf05325ed9b0f1d2c6e70455b0885ca68ec29c9 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Fri, 1 Jan 2021 15:12:50 -0800 Subject: [PATCH 045/105] Reorganize history_item_t Move the private bits to the bottom of the class and other mild refactoring. No user visible behavior change expected. --- src/history.h | 51 ++++++++++++++++++++++---------------------- src/history_file.cpp | 2 +- 2 files changed, 27 insertions(+), 26 deletions(-) diff --git a/src/history.h b/src/history.h index 5f9f16693..0f06f4aa1 100644 --- a/src/history.h +++ b/src/history.h @@ -63,10 +63,28 @@ enum class history_search_type_t { typedef uint64_t history_identifier_t; class history_item_t { - friend class history_t; - friend struct history_impl_t; - friend class history_lru_cache_t; - friend class history_tests_t; + public: + /// Construct from a text, optional timestamp, and optional identifier. + explicit history_item_t(wcstring str = wcstring(), time_t when = 0, + history_identifier_t ident = 0); + + /// \return the text as a string. + const wcstring &str() const { return contents; } + + /// \return whether the text is empty. + bool empty() const { return contents.empty(); } + + // \return wehther our contents matches a search term. + bool matches_search(const wcstring &term, enum history_search_type_t type, + bool case_sensitive) const; + + /// \return the timestamp for creating this history item. + time_t timestamp() const { return creation_timestamp; } + + /// Get and set the list of arguments which referred to files. + /// This is used for autosuggestion hinting. + const path_list_t &get_required_paths() const { return required_paths; } + void set_required_paths(path_list_t paths) { required_paths = std::move(paths); } private: // Attempts to merge two compatible history items together. @@ -84,27 +102,10 @@ class history_item_t { // Paths that we require to be valid for this item to be autosuggested. path_list_t required_paths; - public: - explicit history_item_t(wcstring str = wcstring(), time_t when = 0, - history_identifier_t ident = 0); - - const wcstring &str() const { return contents; } - - bool empty() const { return contents.empty(); } - - // Whether our contents matches a search term. - bool matches_search(const wcstring &term, enum history_search_type_t type, - bool case_sensitive) const; - - time_t timestamp() const { return creation_timestamp; } - - const path_list_t &get_required_paths() const { return required_paths; } - void set_required_paths(const path_list_t &paths) { required_paths = paths; } - - bool operator==(const history_item_t &other) const { - return contents == other.contents && creation_timestamp == other.creation_timestamp && - required_paths == other.required_paths; - } + friend class history_t; + friend struct history_impl_t; + friend class history_lru_cache_t; + friend class history_tests_t; }; typedef std::deque history_item_list_t; diff --git a/src/history_file.cpp b/src/history_file.cpp index 75b905bf6..a3a9e8aed 100644 --- a/src/history_file.cpp +++ b/src/history_file.cpp @@ -280,7 +280,7 @@ static history_item_t decode_item_fish_2_0(const char *base, size_t len) { done: history_item_t result(cmd, when); - result.set_required_paths(paths); + result.set_required_paths(std::move(paths)); return result; } From 9fdc4f903b8b421b18389a0f290d72cc88c128bb Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Sat, 2 Jan 2021 14:34:19 -0800 Subject: [PATCH 046/105] Explicitly track persistence mode in history_item_t Commands that start with a space should not be written to the history file. Prior to this change, that was implemented by simply not adding them to history. Items with leading spaces were simply dropped. With this change, we add a 'history_persistence_mode_t' to history_item_t, which tracks how the item persists. Items with leading spaces are now marked as "ephemeral": they can be recovered via up arrow, until the user runs another command, or types a space and hits return. This matches zsh's HIST_IGNORE_SPACE feature. Fixes #1383 --- src/fish_tests.cpp | 2 +- src/history.cpp | 173 ++++++++++++++++++++++++++----------------- src/history.h | 47 ++++++++---- src/history_file.cpp | 1 + src/reader.cpp | 56 ++++++++------ 5 files changed, 174 insertions(+), 105 deletions(-) diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 0a2153a18..379a475bc 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -4406,7 +4406,7 @@ static bool history_equals(history_t &hist, const wchar_t *const *strings) { history_item_t item = hist.item_at_index(history_idx); if (expected == NULL) { if (!item.empty()) { - err(L"Expected empty item at history index %lu", history_idx); + err(L"Expected empty item at history index %lu, instead found: %ls", history_idx, item.str().c_str()); } break; } else { diff --git a/src/history.cpp b/src/history.cpp index 0f742907f..1963e8cfc 100644 --- a/src/history.cpp +++ b/src/history.cpp @@ -172,22 +172,28 @@ class history_lru_cache_t : public lru_cache_tcontents == item.contents) { - this->creation_timestamp = std::max(this->creation_timestamp, item.creation_timestamp); - if (this->required_paths.size() < item.required_paths.size()) { - this->required_paths = item.required_paths; - } - if (this->identifier < item.identifier) { - this->identifier = item.identifier; - } - result = true; + // We can only merge items if they agree on their text and persistence mode. + if (this->contents != item.contents || this->persist_mode != item.persist_mode) { + return false; } - return result; + + // Ok, merge this item. + this->creation_timestamp = std::max(this->creation_timestamp, item.creation_timestamp); + if (this->required_paths.size() < item.required_paths.size()) { + this->required_paths = item.required_paths; + } + if (this->identifier < item.identifier) { + this->identifier = item.identifier; + } + return true; } -history_item_t::history_item_t(wcstring str, time_t when, history_identifier_t ident) - : contents(trim(std::move(str))), creation_timestamp(when), identifier(ident) {} +history_item_t::history_item_t(wcstring str, time_t when, history_identifier_t ident, + history_persistence_mode_t persist_mode) + : contents(std::move(str)), + creation_timestamp(when), + identifier(ident), + persist_mode(persist_mode) {} bool history_item_t::matches_search(const wcstring &term, enum history_search_type_t type, bool case_sensitive) const { @@ -228,9 +234,11 @@ bool history_item_t::matches_search(const wcstring &term, enum history_search_ty } struct history_impl_t { - // Privately add an item. If pending, the item will not be returned by history searches until a - // call to resolve_pending. - void add(const history_item_t &item, bool pending = false, bool do_save = true); + // Add a new history item to the end. If pending is set, the item will not be returned by + // item_at_index until a call to resolve_pending(). Pending items are tracked with an offset + // into the array of new items, so adding a non-pending item has the effect of resolving all + // pending items. + void add(history_item_t item, bool pending = false, bool do_save = true); // Internal function. void clear_file_state(); @@ -266,7 +274,10 @@ struct history_impl_t { // the boundary are considered "old". Items whose timestemps are > the boundary are new, and are // ignored by this instance (unless they came from this instance). The timestamp may be adjusted // by incorporate_external_changes(). - time_t boundary_timestamp{time(nullptr)}; + time_t boundary_timestamp{}; + + /// The most recent "unique" identifier for a history item. + history_identifier_t last_identifier{0}; // How many items we add until the next vacuum. Initially a random value. int countdown_to_vacuum{-1}; @@ -277,6 +288,12 @@ struct history_impl_t { // List of old items, as offsets into out mmap data. std::deque old_item_offsets{}; + /// \return a timestamp for new items - see the implementation for a subtlety. + time_t timestamp_now() const; + + /// \return a new item identifier, incrementing our counter. + history_identifier_t next_identifier() { return ++last_identifier; } + // Figure out the offsets of our file contents. void populate_from_file_contents(); @@ -286,6 +303,11 @@ struct history_impl_t { // Deletes duplicates in new_items. void compact_new_items(); + // Removes trailing ephemeral items. + // Ephemeral items have leading spaces, and can only be retrieved immediately; adding any item + // removes them. + void remove_ephemeral_items(); + // Attempts to rewrite the existing file to a target temporary file // Returns false on error, true on success bool rewrite_to_temporary_file(int existing_fd, int dst_fd) const; @@ -302,7 +324,9 @@ struct history_impl_t { // Saves history unless doing so is disabled. void save_unless_disabled(); - explicit history_impl_t(wcstring name) : name(std::move(name)) {} + explicit history_impl_t(wcstring name) + : name(std::move(name)), boundary_timestamp(time(nullptr)) {} + history_impl_t(history_impl_t &&) = default; ~history_impl_t() = default; @@ -313,20 +337,9 @@ struct history_impl_t { // require populating the history. bool is_empty(); - // Add a new history item to the end. If pending is set, the item will not be returned by - // item_at_index until a call to resolve_pending(). Pending items are tracked with an offset - // into the array of new items, so adding a non-pending item has the effect of resolving all - // pending items. - void add(const wcstring &str, history_identifier_t ident = 0, bool pending = false, - bool save = true); - // Remove a history item. void remove(const wcstring &str); - // Add a new pending history item to the end, and then begin file detection on the items to - // determine which arguments are paths - void add_pending_with_file_detection(const wcstring &str, const wcstring &working_dir_slash); - // Resolves any pending history items, so that they may be returned in history searches. void resolve_pending(); @@ -366,12 +379,14 @@ struct history_impl_t { size_t size(); }; -void history_impl_t::add(const history_item_t &item, bool pending, bool do_save) { +void history_impl_t::add(history_item_t item, bool pending, bool do_save) { + assert(item.timestamp() != 0 && "Should not add an item with a 0 timestamp"); // We use empty items as sentinels to indicate the end of history. // Do not allow them to be added (#6032). if (item.contents.empty()) { return; } + // Try merging with the last item. if (!new_items.empty() && new_items.back().merge(item)) { // We merged, so we don't have to add anything. Maybe this item was pending, but it just got @@ -422,20 +437,6 @@ void history_impl_t::save_unless_disabled() { countdown_to_vacuum--; } -void history_impl_t::add(const wcstring &str, history_identifier_t ident, bool pending, - bool do_save) { - time_t when = time(nullptr); - // Big hack: do not allow timestamps equal to our boundary date. This is because we include - // items whose timestamps are equal to our boundary when reading old history, so we can catch - // "just closed" items. But this means that we may interpret our own items, that we just wrote, - // as old items, if we wrote them in the same second as our birthdate. - if (when == this->boundary_timestamp) { - when++; - } - - this->add(history_item_t(str, when, ident), pending, do_save); -} - // Remove matching history entries from our list of new items. This only supports literal, // case-sensitive, matches. void history_impl_t::remove(const wcstring &str_to_remove) { @@ -557,6 +558,18 @@ std::unordered_map history_impl_t::items_at_indexes(const std::v return result; } +time_t history_impl_t::timestamp_now() const { + time_t when = time(nullptr); + // Big hack: do not allow timestamps equal to our boundary date. This is because we include + // items whose timestamps are equal to our boundary when reading old history, so we can catch + // "just closed" items. But this means that we may interpret our own items, that we just wrote, + // as old items, if we wrote them in the same second as our birthdate. + if (when == this->boundary_timestamp) { + when++; + } + return when; +} + void history_impl_t::populate_from_file_contents() { old_item_offsets.clear(); if (file_contents) { @@ -648,12 +661,15 @@ void history_impl_t::clear_file_state() { } void history_impl_t::compact_new_items() { - // Keep only the most recent items with the given contents. This algorithm could be made more - // efficient, but likely would consume more memory too. + // Keep only the most recent items with the given contents. std::unordered_set seen; size_t idx = new_items.size(); while (idx--) { const history_item_t &item = new_items[idx]; + + // Only compact persisted items. + if (!item.should_write_to_disk()) continue; + if (!seen.insert(item.contents).second) { // This item was not inserted because it was already in the set, so delete the item at // this index. @@ -668,6 +684,14 @@ void history_impl_t::compact_new_items() { } } +void history_impl_t::remove_ephemeral_items() { + while (!new_items.empty() && + new_items.back().persist_mode == history_persistence_mode_t::ephemeral) { + new_items.pop_back(); + } + first_unwritten_new_item_index = std::min(first_unwritten_new_item_index, new_items.size()); +} + // Given the fd of an existing history file, or -1 if none, write // a new history file to temp_fd. Returns true on success, false // on error @@ -698,7 +722,9 @@ bool history_impl_t::rewrite_to_temporary_file(int existing_fd, int dst_fd) cons // Insert any unwritten new items for (auto iter = new_items.cbegin() + this->first_unwritten_new_item_index; iter != new_items.cend(); ++iter) { - lru.add_item(*iter); + if (iter->should_write_to_disk()) { + lru.add_item(*iter); + } } // Stable-sort our items by timestamp @@ -920,10 +946,12 @@ bool history_impl_t::save_internal_via_appending() { std::string buffer; while (first_unwritten_new_item_index < new_items.size()) { const history_item_t &item = new_items.at(first_unwritten_new_item_index); - append_history_item_to_buffer(item, &buffer); - err = flush_to_fd(&buffer, history_fd.fd(), HISTORY_OUTPUT_BUFFER_SIZE); - if (err) break; - // We wrote this item, hooray. + if (item.should_write_to_disk()) { + append_history_item_to_buffer(item, &buffer); + err = flush_to_fd(&buffer, history_fd.fd(), HISTORY_OUTPUT_BUFFER_SIZE); + if (err) break; + } + // We wrote or skipped this item, hooray. first_unwritten_new_item_index++; } @@ -1057,6 +1085,12 @@ bool history_impl_t::is_empty() { return empty; } +void history_t::add(wcstring str) { + auto imp = this->impl(); + time_t when = imp->timestamp_now(); + imp->add(history_item_t(std::move(str), when)); +} + /// Populates from older location (in config path, rather than data path) This is accomplished by /// clearing ourselves, and copying the contents of the old history file to the new history file. /// The new contents will automatically be re-mapped later. @@ -1132,6 +1166,8 @@ static bool should_import_bash_history_line(const wcstring &line) { /// encode multiline commands. void history_impl_t::populate_from_bash(FILE *stream) { // Process the entire history file until EOF is observed. + // Pretend all items were created at this time. + const auto when = this->timestamp_now(); bool eof = false; while (!eof) { auto line = std::string(); @@ -1151,10 +1187,11 @@ void history_impl_t::populate_from_bash(FILE *stream) { if (a_newline) break; } - wcstring wide_line = str2wcstring(line); + wcstring wide_line = trim(str2wcstring(line)); // Add this line if it doesn't contain anything we know we can't handle. if (should_import_bash_history_line(wide_line)) { - this->add(wide_line, 0, false /* pending */, false /* do_save */); + this->add(history_item_t(std::move(wide_line), when), false /* pending */, + false /* do_save */); } } this->save_unless_disabled(); @@ -1163,7 +1200,7 @@ void history_impl_t::populate_from_bash(FILE *stream) { void history_impl_t::incorporate_external_changes() { // To incorporate new items, we simply update our timestamp to now, so that items from previous // instances get added. We then clear the file state so that we remap the file. Note that this - // is somehwhat expensive because we will be going back over old items. An optimization would be + // is somewhat expensive because we will be going back over old items. An optimization would be // to preserve old_item_offsets so that they don't have to be recomputed. (However, then items // *deleted* in other instances would not show up here). time_t new_timestamp = time(nullptr); @@ -1174,9 +1211,11 @@ void history_impl_t::incorporate_external_changes() { this->boundary_timestamp = new_timestamp; this->clear_file_state(); - // We also need to erase new_items, since we go through those first, and that means we + // We also need to erase new items, since we go through those first, and that means we // will not properly interleave them with items from other instances. // We'll pick them up from the file (#2312) + // TODO: this will drop items that had no_persist set, how can we avoid that while still + // properly interleaving? this->save(false); this->new_items.clear(); this->first_unwritten_new_item_index = 0; @@ -1258,16 +1297,15 @@ bool history_t::is_default() const { return impl()->is_default(); } bool history_t::is_empty() { return impl()->is_empty(); } -void history_t::add(const history_item_t &item, bool pending) { impl()->add(item, pending); } - -void history_t::add(const wcstring &str, history_identifier_t ident, bool pending) { - impl()->add(str, ident, pending); -} +void history_t::add(history_item_t item, bool pending) { impl()->add(std::move(item), pending); } void history_t::remove(const wcstring &str) { impl()->remove(str); } +void history_t::remove_ephemeral_items() { impl()->remove_ephemeral_items(); } + void history_t::add_pending_with_file_detection(const wcstring &str, - const wcstring &working_dir_slash) { + const wcstring &working_dir_slash, + history_persistence_mode_t persist_mode) { // We use empty items as sentinels to indicate the end of history. // Do not allow them to be added (#6032). if (str.empty()) { @@ -1311,17 +1349,18 @@ void history_t::add_pending_with_file_detection(const wcstring &str, bool wants_file_detection = !potential_paths.empty() && !needs_sync_write; auto imp = this->impl(); - history_identifier_t identifier = 0; + // Make our history item. + time_t when = imp->timestamp_now(); + history_identifier_t identifier = imp->next_identifier(); + history_item_t item{str, when, identifier, persist_mode}; + if (wants_file_detection) { - // Grab the next identifier. - static relaxed_atomic_t s_last_identifier{0}; - identifier = ++s_last_identifier; imp->disable_automatic_saving(); // Add the item. Then check for which paths are valid on a background thread, // and unblock the item. // Don't hold the lock while we perform this file detection. - imp->add(str, identifier, true /* pending */); + imp->add(std::move(item), true /* pending */); iothread_perform([=]() { auto validated_paths = valid_paths(potential_paths, working_dir_slash); auto imp = this->impl(); @@ -1332,7 +1371,7 @@ void history_t::add_pending_with_file_detection(const wcstring &str, // Add the item. // If we think we're about to exit, save immediately, regardless of any disabling. This may // cause us to lose file hinting for some commands, but it beats losing history items. - imp->add(str, identifier, true /* pending */); + imp->add(std::move(item), true /* pending */); if (needs_sync_write) { imp->save(); } diff --git a/src/history.h b/src/history.h index 0f06f4aa1..053a4b0d9 100644 --- a/src/history.h +++ b/src/history.h @@ -62,11 +62,20 @@ enum class history_search_type_t { typedef uint64_t history_identifier_t; +/// Ways that a history item may be written to disk (or omitted). +enum class history_persistence_mode_t : uint8_t { + disk, // the history item is written to disk normally + memory, // the history item is stored in-memory only, not written to disk + ephemeral, // the history item is stored in-memory and deleted when a new item is added +}; + class history_item_t { public: - /// Construct from a text, optional timestamp, and optional identifier. - explicit history_item_t(wcstring str = wcstring(), time_t when = 0, - history_identifier_t ident = 0); + /// Construct from a text, timestamp, and optional identifier. + /// If \p no_persist is set, then do not write this item to disk. + explicit history_item_t( + wcstring str = {}, time_t when = 0, history_identifier_t ident = 0, + history_persistence_mode_t persist_mode = history_persistence_mode_t::disk); /// \return the text as a string. const wcstring &str() const { return contents; } @@ -81,6 +90,9 @@ class history_item_t { /// \return the timestamp for creating this history item. time_t timestamp() const { return creation_timestamp; } + /// \return whether this item should be persisted (written to disk). + bool should_write_to_disk() const { return persist_mode == history_persistence_mode_t::disk; } + /// Get and set the list of arguments which referred to files. /// This is used for autosuggestion hinting. const path_list_t &get_required_paths() const { return required_paths; } @@ -96,11 +108,14 @@ class history_item_t { // Original creation time for the entry. time_t creation_timestamp; + // Paths that we require to be valid for this item to be autosuggested. + path_list_t required_paths; + // Sometimes unique identifier used for hinting. history_identifier_t identifier; - // Paths that we require to be valid for this item to be autosuggested. - path_list_t required_paths; + // If set, do not write this item to disk. + history_persistence_mode_t persist_mode; friend class history_t; friend struct history_impl_t; @@ -129,8 +144,11 @@ class history_t { acquired_lock impl() const; // Privately add an item. If pending, the item will not be returned by history searches until a - // call to resolve_pending. - void add(const history_item_t &item, bool pending = false); + // call to resolve_pending. Any trailing ephemeral items are dropped. + void add(history_item_t item, bool pending = false); + + // Add a new history item with text \p str to the end of history. + void add(wcstring str); public: explicit history_t(wcstring name); @@ -153,18 +171,17 @@ class history_t { // require populating the history. bool is_empty(); - // Add a new history item to the end. If pending is set, the item will not be returned by - // item_at_index until a call to resolve_pending(). Pending items are tracked with an offset - // into the array of new items, so adding a non-pending item has the effect of resolving all - // pending items. - void add(const wcstring &str, history_identifier_t ident = 0, bool pending = false); - // Remove a history item. void remove(const wcstring &str); + /// Remove any trailing ephemeral items. + void remove_ephemeral_items(); + // Add a new pending history item to the end, and then begin file detection on the items to - // determine which arguments are paths - void add_pending_with_file_detection(const wcstring &str, const wcstring &working_dir_slash); + // determine which arguments are paths. The item has the given \p persist_mode. + void add_pending_with_file_detection( + const wcstring &str, const wcstring &working_dir_slash, + history_persistence_mode_t persist_mode = history_persistence_mode_t::disk); // Resolves any pending history items, so that they may be returned in history searches. void resolve_pending(); diff --git a/src/history_file.cpp b/src/history_file.cpp index a3a9e8aed..7598191c1 100644 --- a/src/history_file.cpp +++ b/src/history_file.cpp @@ -431,6 +431,7 @@ static size_t offset_of_next_item_fish_2_0(const history_file_contents_t &conten } void append_history_item_to_buffer(const history_item_t &item, std::string *buffer) { + assert(item.should_write_to_disk() && "Item should not be persisted"); auto append = [=](const char *a, const char *b = nullptr, const char *c = nullptr) { if (a) buffer->append(a); if (b) buffer->append(b); diff --git a/src/reader.cpp b/src/reader.cpp index cf81c6710..a9f7d855f 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -1576,10 +1576,6 @@ void reader_data_t::completion_insert(const wcstring &val, size_t token_end, set_buffer_maintaining_pager(new_command_line, cursor); } -static bool may_add_to_history(const wcstring &commandline_prefix) { - return !commandline_prefix.empty() && commandline_prefix.at(0) != L' '; -} - // Returns a function that can be invoked (potentially // on a background thread) to determine the autosuggestion static std::function get_autosuggestion_performer( @@ -1603,21 +1599,20 @@ static std::function get_autosuggestion_performer( return nothing; } - if (may_add_to_history(search_string)) { - history_search_t searcher(*history, search_string, history_search_type_t::prefix, - history_search_flags_t{}); - while (!ctx.check_cancel() && searcher.go_backwards()) { - const history_item_t &item = searcher.current_item(); + // Search history for a matching item. + history_search_t searcher(*history, search_string, history_search_type_t::prefix, + history_search_flags_t{}); + while (!ctx.check_cancel() && searcher.go_backwards()) { + const history_item_t &item = searcher.current_item(); - // Skip items with newlines because they make terrible autosuggestions. - if (item.str().find(L'\n') != wcstring::npos) continue; + // Skip items with newlines because they make terrible autosuggestions. + if (item.str().find(L'\n') != wcstring::npos) continue; - if (autosuggest_validate_from_history(item, working_directory, ctx)) { - // The command autosuggestion was handled specially, so we're done. - // History items are case-sensitive, see #3978. - return autosuggestion_t{searcher.current_string(), search_string, - false /* icase */}; - } + if (autosuggest_validate_from_history(item, working_directory, ctx)) { + // The command autosuggestion was handled specially, so we're done. + // History items are case-sensitive, see #3978. + return autosuggestion_t{searcher.current_string(), search_string, + false /* icase */}; } } @@ -3158,11 +3153,28 @@ void reader_data_t::handle_readline_command(readline_cmd_t c, readline_loop_stat } if (command_test_result == 0) { - // Finished command, execute it. Don't add items that start with a leading - // space, or if in silent mode (#7230). - const editable_line_t *el = &command_line; - if (history != nullptr && !conf.in_silent_mode && may_add_to_history(el->text())) { - history->add_pending_with_file_detection(el->text(), vars.get_pwd_slash()); + // Finished command, execute it. Don't add items in silent mode (#7230). + wcstring text = command_line.text(); + if (text.empty()) { + // Here the user just hit return. Make a new prompt, don't remove ephemeral + // items. + rls.finished = true; + break; + } + + // Historical behavior is to trim trailing spaces. + while (!text.empty() && text.back() == L' ') { + text.pop_back(); + } + if (history && !conf.in_silent_mode) { + // Remove ephemeral items. + // Note we fall into this case if the user just types a space and hits return. + history->remove_ephemeral_items(); + + // Mark this item as ephemeral if there is a leading space (#615). + auto mode = text.front() == L' ' ? history_persistence_mode_t::ephemeral + : history_persistence_mode_t::disk; + history->add_pending_with_file_detection(text, vars.get_pwd_slash(), mode); } rls.finished = true; update_buff_pos(&command_line, command_line.size()); From 118f710e996e0ddadaff9906cc8b76aabdd6a115 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Fri, 1 Jan 2021 13:57:45 -0800 Subject: [PATCH 047/105] Allow fish_private_mode to change at runtime Prior to this change, `fish_private_mode` worked by just suppressing history outright. With this change, `fish_private_mode` can be toggled on and off. Commands entered while `fish_private_mode` is set are stored but in memory only; they are not written to disk. Fixes #7590 Fixes #7589 --- doc_src/index.rst | 6 ++- src/env.cpp | 4 +- src/history.cpp | 5 ++- src/history.h | 3 +- src/reader.cpp | 14 ++++++- tests/pexpects/private_mode.py | 70 ++++++++++++++++++++++++++++++++++ 6 files changed, 93 insertions(+), 9 deletions(-) create mode 100644 tests/pexpects/private_mode.py diff --git a/doc_src/index.rst b/doc_src/index.rst index dd98587a9..23d38b3e3 100644 --- a/doc_src/index.rst +++ b/doc_src/index.rst @@ -2003,7 +2003,11 @@ If a function named :ref:`fish_greeting ` exists, it will be Private mode ------------- -fish supports launching in private mode via ``fish --private`` (or ``fish -P`` for short). In private mode, old history is not available and any interactive commands you execute will not be appended to the global history file, making it useful both for avoiding inadvertently leaking personal information (e.g. for screencasts) and when dealing with sensitive information to prevent it being persisted to disk. You can query the global variable ``fish_private_mode`` (``if set -q fish_private_mode ...``) if you would like to respect the user's wish for privacy and alter the behavior of your own fish scripts. +If ``$fish_private_mode`` is set to a non-empty value, commands will not be written to the history file on disk. + +You can also launch with ``fish --private`` (or ``fish -P`` for short). This both hides old history and prevents writing history to disk. This is useful to avoid leaking personal information (e.g. for screencasts) or when dealing with sensitive information. + +You can query the variable ``fish_private_mode`` (``if set -q fish_private_mode ...``) if you would like to respect the user's wish for privacy and alter the behavior of your own fish scripts. .. _event: diff --git a/src/env.cpp b/src/env.cpp index e55159120..80fadb131 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -90,7 +90,6 @@ static const std::vector electric_variables{ {L"_", electric_var_t::freadonly}, {L"fish_kill_signal", electric_var_t::freadonly | electric_var_t::fcomputed}, {L"fish_pid", electric_var_t::freadonly}, - {L"fish_private_mode", electric_var_t::freadonly}, {L"history", electric_var_t::freadonly | electric_var_t::fcomputed}, {L"hostname", electric_var_t::freadonly}, {L"pipestatus", electric_var_t::freadonly | electric_var_t::fcomputed}, @@ -119,8 +118,7 @@ static bool is_read_only(const wcstring &key) { if (auto ev = electric_var_t::for_name(key)) { return ev->readonly(); } - // Hack. - return in_private_mode() && key == L"fish_history"; + return false; } /// Return true if a variable should become a path variable by default. See #436. diff --git a/src/history.cpp b/src/history.cpp index 1963e8cfc..1ae664cb5 100644 --- a/src/history.cpp +++ b/src/history.cpp @@ -1485,9 +1485,10 @@ history_t &history_t::history_with_name(const wcstring &name) { static relaxed_atomic_bool_t private_mode{false}; void start_private_mode(env_stack_t &vars) { - private_mode = true; vars.set_one(L"fish_history", ENV_GLOBAL, L""); vars.set_one(L"fish_private_mode", ENV_GLOBAL, L"1"); } -bool in_private_mode() { return private_mode; } +bool in_private_mode(const environment_t &vars) { + return !vars.get(L"fish_private_mode").missing_or_empty(); +} diff --git a/src/history.h b/src/history.h index 053a4b0d9..7506f62ab 100644 --- a/src/history.h +++ b/src/history.h @@ -312,7 +312,8 @@ bool all_paths_are_valid(const path_list_t &paths, const wcstring &working_direc /// Sets private mode on. Once in private mode, it cannot be turned off. void start_private_mode(env_stack_t &vars); + /// Queries private mode status. -bool in_private_mode(); +bool in_private_mode(const environment_t &vars); #endif diff --git a/src/reader.cpp b/src/reader.cpp index a9f7d855f..eb50ffcba 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -3166,16 +3166,26 @@ void reader_data_t::handle_readline_command(readline_cmd_t c, readline_loop_stat while (!text.empty() && text.back() == L' ') { text.pop_back(); } + if (history && !conf.in_silent_mode) { // Remove ephemeral items. // Note we fall into this case if the user just types a space and hits return. history->remove_ephemeral_items(); // Mark this item as ephemeral if there is a leading space (#615). - auto mode = text.front() == L' ' ? history_persistence_mode_t::ephemeral - : history_persistence_mode_t::disk; + history_persistence_mode_t mode; + if (text.front() == L' ') { + // Leading spaces are ephemeral (#615). + mode = history_persistence_mode_t::ephemeral; + } else if (in_private_mode(vars)) { + // Private mode means in-memory only. + mode = history_persistence_mode_t::memory; + } else { + mode = history_persistence_mode_t::disk; + } history->add_pending_with_file_detection(text, vars.get_pwd_slash(), mode); } + rls.finished = true; update_buff_pos(&command_line, command_line.size()); } else if (command_test_result == PARSER_TEST_INCOMPLETE) { diff --git a/tests/pexpects/private_mode.py b/tests/pexpects/private_mode.py new file mode 100644 index 000000000..239264e65 --- /dev/null +++ b/tests/pexpects/private_mode.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +import os +import time +from pexpect_helper import SpawnedProc + +sp = SpawnedProc() +sendline, sleep, expect_prompt, expect_str = ( + sp.sendline, + sp.sleep, + sp.expect_prompt, + sp.expect_str, +) + +# Helper to sendline and add to our view of history. +recorded_history = [] +private_mode_active = False +fish_path = os.environ.get("fish") + +# Send a line and record it in our history array if private mode is not active. +def sendline_record(s): + sendline(s) + if not private_mode_active: + recorded_history.append(s) + + +expect_prompt() + +# Start off with no history. +sendline(r" builtin history clear; builtin history save") +expect_prompt() + +# Ensure that fish_private_mode can be changed - see #7589. +sendline_record(r"echo before_private_mode") +expect_prompt("before_private_mode") +sendline(r" builtin history save") +expect_prompt() + +# Enter private mode. +sendline_record(r"set -g fish_private_mode 1") +expect_prompt() +private_mode_active = True + +sendline_record(r"echo check2 $fish_private_mode") +expect_prompt("check2 1") + +# Nothing else gets added. +sendline_record(r"true") +expect_prompt() +sendline_record(r"false") +expect_prompt() + +# Leave private mode. The command to leave it is still private. +sendline_record(r"set -ge fish_private_mode") +expect_prompt() +private_mode_active = False + +# New commands get added. +sendline_record(r"set alpha beta") +expect_prompt() + +# Check our history is what we expect. +# We have to wait for the time to tick over, else our item risks being discarded. +now = time.time() +start = int(now) +while now - start < 1: + sleep(now - start) + now = time.time() + +sendline(r" builtin history save ; %s -c 'string join \n -- $history'" % fish_path) +expect_prompt("\r\n".join(reversed(recorded_history))) From 29121ffc4cf6128b8cb6a7a54d51b2f60dc246f5 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Sat, 2 Jan 2021 22:18:14 -0800 Subject: [PATCH 048/105] Relnote fixes for #7589 and #1383 --- CHANGELOG.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index b056cff82..68ddea838 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -153,6 +153,8 @@ Interactive improvements - ``fish_trace`` output now starts with ``->`` like ``fish --profile``'s, making the depth more visible (:issue:`7538`). - Resizing the terminal window no longer produces a corrupted prompt (:issue:`6532`). - ``functions`` produces an error rather than crashing on certain invalid arguments (:issue:`7515`). +- ``fish_private_mode`` may now be changed dynamically using ``set`` (:issue:`7589`). +- Commands with leading spaces may be retrieved from history with up-arrow until a new command is run, matching zsh's ``HIST_IGNORE_SPACE`` (:issue:`1383`). New or improved bindings ^^^^^^^^^^^^^^^^^^^^^^^^ From 18940ea086ebff326e9e76a53430cabe7f9a69b7 Mon Sep 17 00:00:00 2001 From: Ilan Cosman Date: Sun, 3 Jan 2021 06:15:57 -0800 Subject: [PATCH 049/105] Remove dunderscores from __fish_status_to_signal (#7597) * Remove dunderscores from __fish_status_to_signal * Document fish_status_to_signal * CHANGELOG: Add fish_status_to_signal * Add string join to fish_status_to_signal documentation example --- CHANGELOG.rst | 1 + doc_src/cmds/fish_status_to_signal.rst | 29 +++++++++++++++++++ share/functions/__fish_print_pipestatus.fish | 2 +- ...signal.fish => fish_status_to_signal.fish} | 2 +- 4 files changed, 32 insertions(+), 2 deletions(-) create mode 100644 doc_src/cmds/fish_status_to_signal.rst rename share/functions/{__fish_status_to_signal.fish => fish_status_to_signal.fish} (85%) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 68ddea838..b02854286 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -102,6 +102,7 @@ Scripting improvements - ``string`` subcommands now quit early when used with ``--quiet`` (:issue:`7495`). - Failed redirections will now set ``$status`` (:issue:`7540`). - ``read`` can now read interactively from other files, so e.g. forcing it to read from the terminal via ``read _ sleep 5 + ^C⏎ + >_ fish_status_to_signal $status + SIGINT \ No newline at end of file diff --git a/share/functions/__fish_print_pipestatus.fish b/share/functions/__fish_print_pipestatus.fish index 22f75def4..aad28909a 100644 --- a/share/functions/__fish_print_pipestatus.fish +++ b/share/functions/__fish_print_pipestatus.fish @@ -23,7 +23,7 @@ function __fish_print_pipestatus --description "Print pipestatus for prompt" # SIGPIPE (141 = 128 + 13) is usually not a failure, see #6375. if not contains $last_status 0 141 set -l sep $brace_sep_color$separator$status_color - set -l last_pipestatus_string (__fish_status_to_signal $argv | string join "$sep") + set -l last_pipestatus_string (fish_status_to_signal $argv | string join "$sep") set -l last_status_string "" if test $last_status -ne $argv[-1] set last_status_string " "$status_color$last_status diff --git a/share/functions/__fish_status_to_signal.fish b/share/functions/fish_status_to_signal.fish similarity index 85% rename from share/functions/__fish_status_to_signal.fish rename to share/functions/fish_status_to_signal.fish index 47fa38ff5..2d09916dc 100644 --- a/share/functions/__fish_status_to_signal.fish +++ b/share/functions/fish_status_to_signal.fish @@ -1,4 +1,4 @@ -function __fish_status_to_signal --description "Print signal name from argument (\$status), or just argument" +function fish_status_to_signal --description "Print signal name from argument (\$status), or just argument" for arg in $argv if test $arg -gt 128 set -l signals SIGHUP SIGINT SIGQUIT SIGILL SIGTRAP SIGABRT SIGBUS \ From e33255559624c0b603a4ccfb7f351d4ecb4bd699 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sat, 2 Jan 2021 16:36:24 +0100 Subject: [PATCH 050/105] Webconfig: Remove dependency on cgi module This is slated for removal in python 3.10, see https://www.python.org/dev/peps/pep-0594/#cgi. We currently only use it for three things: - escape_html in old python versions that didn't have that in the html module - Parsing multipart/form-data - Figuring out the charset for json We keep the first one - if loading escape_html from html fails we fall back to cgi. We remove the second - I can't find any case where we use multipart/form-data. Any place we post data we either explicitly pass application/x-www-form-urlencoded or implicitly use application/json. The third is the tricky bit. This drops charset detection under the assumption that we're never going to encounter anything other than utf-8 (or ascii, which is a utf-8 subset). I'm not sure that holds, but if it doesn't we can just add a regex to parse the charset. --- share/tools/web_config/webconfig.py | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/share/tools/web_config/webconfig.py b/share/tools/web_config/webconfig.py index a46aacbaa..c6529ea26 100755 --- a/share/tools/web_config/webconfig.py +++ b/share/tools/web_config/webconfig.py @@ -1,7 +1,6 @@ from __future__ import unicode_literals from __future__ import print_function import binascii -import cgi try: from html import escape as escape_html @@ -1315,17 +1314,19 @@ class FishConfigHTTPRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler): return self.send_error(403) self.path = p - ctype, pdict = cgi.parse_header(self.headers["content-type"]) + # This is cheesy, we want just the actual content-type. + # In some cases it'll give us the encoding as well, + # ("application/json;charset=utf-8") + # but we don't currently care. + ctype = self.headers["content-type"].split(";")[0] - if ctype == "multipart/form-data": - postvars = cgi.parse_multipart(self.rfile, pdict) - elif ctype == "application/x-www-form-urlencoded": + if ctype == "application/x-www-form-urlencoded": length = int(self.headers["content-length"]) url_str = self.rfile.read(length).decode("utf-8") postvars = parse_qs(url_str, keep_blank_values=1) elif ctype == "application/json": length = int(self.headers["content-length"]) - url_str = self.rfile.read(length).decode(pdict["charset"]) + url_str = self.rfile.read(length).decode("utf-8") postvars = json.loads(url_str) else: postvars = {} From cb3ee51e085274bc6fd48504e021474fb678a5ee Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sun, 3 Jan 2021 15:18:15 +0100 Subject: [PATCH 051/105] CHANGELOG cgi removal --- CHANGELOG.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index b02854286..6e068d68c 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -303,7 +303,7 @@ For distributors and developers codesigning is enabled (:issue:`6952`). - Running the full interactive test suite now requires Python 3.5+ and the pexpect package (:issue:`6825`); the expect package is no longer required. - Support for Python 2 in fish's tools (``fish_config`` and the manual page completion generator) is no longer guaranteed. Please use Python 3.5 or later (:issue:`6537`). -- The web-based configuration tool no longer requires Python's distutils (:issue:`7514`) +- The web-based configuration tool no longer requires Python's distutils (:issue:`7514`) or the deprecated cgi module (:issue:`7600`). - fish 3.2 is the last release to support Red Hat Enterprise Linux & CentOS version 7. -------------- From 17501bcc5750574c66722cde2560ae9643b97e3a Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sun, 3 Jan 2021 15:27:46 +0100 Subject: [PATCH 052/105] webconfig: Error out on form-data Just in case this happens anywhere return a sensible error instead of mishandling it. --- share/tools/web_config/webconfig.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/share/tools/web_config/webconfig.py b/share/tools/web_config/webconfig.py index c6529ea26..c8c3c7807 100755 --- a/share/tools/web_config/webconfig.py +++ b/share/tools/web_config/webconfig.py @@ -1328,6 +1328,10 @@ class FishConfigHTTPRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler): length = int(self.headers["content-length"]) url_str = self.rfile.read(length).decode("utf-8") postvars = json.loads(url_str) + elif ctype == "multipart/form-data": + # This used to be a thing, as far as I could find there's + # no use anymore, but let's keep an error around just in case. + return self.send_error(500) else: postvars = {} From 627fff797198d2d3ad3d1207b03a89dd810c4d3d Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sun, 3 Jan 2021 15:43:12 +0100 Subject: [PATCH 053/105] webconfig: Comment utf-8 assumption --- share/tools/web_config/webconfig.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/share/tools/web_config/webconfig.py b/share/tools/web_config/webconfig.py index c8c3c7807..01c7b7aaf 100755 --- a/share/tools/web_config/webconfig.py +++ b/share/tools/web_config/webconfig.py @@ -1326,6 +1326,12 @@ class FishConfigHTTPRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler): postvars = parse_qs(url_str, keep_blank_values=1) elif ctype == "application/json": length = int(self.headers["content-length"]) + # This used to use the provided encoding, but we use utf-8 + # all around the place and nobody has ever complained. + # + # If any other encoding is received this will raise a UnicodeError, + # which will throw us out of the function and should at most exit webconfig. + # If that happens to anyone we expect bug reports. url_str = self.rfile.read(length).decode("utf-8") postvars = json.loads(url_str) elif ctype == "multipart/form-data": From 85ba2ed790b247b79acde136a2c640417bc140e0 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sun, 3 Jan 2021 17:41:38 +0100 Subject: [PATCH 054/105] type: Add missing newline Otherwise this would print # Defined interactivelyfunction foo for interactively defined functions. --- src/builtin_type.cpp | 2 +- tests/pexpects/generic.py | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/src/builtin_type.cpp b/src/builtin_type.cpp index cdd64c3df..3a1fa3ec8 100644 --- a/src/builtin_type.cpp +++ b/src/builtin_type.cpp @@ -155,7 +155,7 @@ maybe_t builtin_type(parser_t &parser, io_streams_t &streams, wchar_t **arg def = comment.append(def); } else { wcstring comment; - append_format(comment, L"# Defined interactively"); + append_format(comment, L"# Defined interactively\n"); def = comment.append(def); } if (!streams.out_is_redirected && isatty(STDOUT_FILENO)) { diff --git a/tests/pexpects/generic.py b/tests/pexpects/generic.py index efff0bcb5..5dd0ed618 100644 --- a/tests/pexpects/generic.py +++ b/tests/pexpects/generic.py @@ -56,3 +56,11 @@ expect_prompt("error: no-execute mode enabled and no script given. Exiting") sendline("source; or echo failed") expect_prompt("failed") + +# See that `type` tells us the function was defined interactively. +sendline("function foo; end; type foo") +expect_str("foo is a function with definition\r\n") +expect_str("# Defined interactively\r\n") +expect_str("function foo") +expect_str("end") +expect_prompt() From 4116aaeb5fc90aa7088f3be608d08dc2a1f66583 Mon Sep 17 00:00:00 2001 From: Weihang Lo Date: Mon, 4 Jan 2021 00:53:33 +0800 Subject: [PATCH 055/105] Update rustc.fish - [`-L`: add a directory to the library search path][1] - [`--crate-type`: a list of types of crates for the compiler to emit][2] - [`--emit`: specifies the types of output files to generate][3] [1]: https://doc.rust-lang.org/stable/rustc/command-line-arguments.html#-l-add-a-directory-to-the-library-search-path [2]: https://doc.rust-lang.org/stable/rustc/command-line-arguments.html#--crate-type-a-list-of-types-of-crates-for-the-compiler-to-emit [3]: https://doc.rust-lang.org/stable/rustc/command-line-arguments.html#--emit-specifies-the-types-of-output-files-to-generate --- share/completions/rustc.fish | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/share/completions/rustc.fish b/share/completions/rustc.fish index 4b2f1649a..0a905bcf6 100644 --- a/share/completions/rustc.fish +++ b/share/completions/rustc.fish @@ -1,11 +1,11 @@ complete -c rustc -s h -l help complete -c rustc -x -l cfg -complete -c rustc -r -s L -a 'dylib= static= framework=' +complete -c rustc -r -s L -a 'dependency= crate= native= framework= all=' complete -c rustc -x -s l -a 'dylib= static= framework=' -complete -c rustc -x -l crate-type -a 'bin lib rlib dylib staticlib' +complete -c rustc -x -l crate-type -a 'bin lib rlib dylib staticlib proc-macro' complete -c rustc -r -l crate-name -complete -c rustc -x -l emit -a 'asm llvm-bc llvm-ir obj link dep-info' +complete -c rustc -x -l emit -a 'asm llvm-bc llvm-ir obj link dep-info metadata mir' complete -c rustc -x -l print -a 'crate-name file-names sysroot' complete -c rustc -s g complete -c rustc -s O From 6eeb8861e78608564b1e2307abd89ecc95f4f751 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Mon, 4 Jan 2021 09:45:34 +0100 Subject: [PATCH 056/105] Add `exit` bind function Currently binding `exit` to a key checks too late that it's exitted, so it leaves the shell hanging around until the user does an execute or similar. As I understand it, the `exit` builtin is supposed to only exit the current "thread" (once that actually becomes a thing), and the bindings would probably run in a dedicated one, so the simplest solution here is to just add an `exit` bind function. Fixes #7604. --- doc_src/cmds/bind.rst | 2 ++ src/input.cpp | 1 + src/input_common.h | 1 + src/reader.cpp | 7 +++++++ 4 files changed, 11 insertions(+) diff --git a/doc_src/cmds/bind.rst b/doc_src/cmds/bind.rst index 4bb2096b7..4c5be95f7 100644 --- a/doc_src/cmds/bind.rst +++ b/doc_src/cmds/bind.rst @@ -126,6 +126,8 @@ The following special input functions are available: - ``execute``, run the current commandline +- ``exit``, exit the shell + - ``forward-bigword``, move one whitespace-delimited word to the right - ``forward-char``, move one character to the right diff --git a/src/input.cpp b/src/input.cpp index bcac349c5..cd3c22eba 100644 --- a/src/input.cpp +++ b/src/input.cpp @@ -152,6 +152,7 @@ static const input_function_metadata_t input_function_metadata[] = { {readline_cmd_t::func_or, L"or"}, {readline_cmd_t::expand_abbr, L"expand-abbr"}, {readline_cmd_t::delete_or_exit, L"delete-or-exit"}, + {readline_cmd_t::exit, L"exit"}, {readline_cmd_t::cancel_commandline, L"cancel-commandline"}, {readline_cmd_t::cancel, L"cancel"}, {readline_cmd_t::undo, L"undo"}, diff --git a/src/input_common.h b/src/input_common.h index 99c5041c9..09fe9dba6 100644 --- a/src/input_common.h +++ b/src/input_common.h @@ -75,6 +75,7 @@ enum class readline_cmd_t { func_or, expand_abbr, delete_or_exit, + exit, cancel_commandline, cancel, undo, diff --git a/src/reader.cpp b/src/reader.cpp index eb50ffcba..42e907d9a 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -3076,6 +3076,13 @@ void reader_data_t::handle_readline_command(readline_cmd_t c, readline_loop_stat delete_char(); break; } + case rl::exit: { + // This is by definition a successful exit, override the status + parser().set_last_statuses(statuses_t::just(STATUS_CMD_OK)); + exit_loop_requested = true; + check_exit_loop_maybe_warning(this); + break; + } case rl::delete_or_exit: case rl::delete_char: { // Remove the current character in the character buffer and on the screen using From 768defeb8ed0c5c4db30c956c225726a5d0a01db Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Mon, 4 Jan 2021 12:23:29 +0100 Subject: [PATCH 057/105] webconfig: Stop proscribing a specific font-family There's a macOS bug with Source Code Pro that makes it unable to be colored. Since that makes webconfig unusable, stop recommending it. Instead, we just pick the default monospace font for the system. --- share/tools/web_config/fishconfig.css | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/share/tools/web_config/fishconfig.css b/share/tools/web_config/fishconfig.css index 5a988c810..18de70a37 100644 --- a/share/tools/web_config/fishconfig.css +++ b/share/tools/web_config/fishconfig.css @@ -1,7 +1,6 @@ body { background: linear-gradient(to bottom, #a7cfdf 0%, #23538a 100%); - font-family: "Source Code Pro", "DejaVu Sans Mono", Menlo, "Ubuntu Mono", Consolas, Monaco, - "Lucida Console", monospace, fixed; + font-family: monospace, fixed; color: #222; min-height: 100vh; /* at least 1 screen high - to prevent the gradient from running out on a short tab */ width: 90%; From 7c704ce5453b251f2f89febbe206a93647a509b8 Mon Sep 17 00:00:00 2001 From: Edouard Lopez Date: Mon, 4 Jan 2021 13:28:58 +0100 Subject: [PATCH 058/105] use original theme repo URL --- share/tools/web_config/js/colorutils.js | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/share/tools/web_config/js/colorutils.js b/share/tools/web_config/js/colorutils.js index ddab52206..720efaa5b 100644 --- a/share/tools/web_config/js/colorutils.js +++ b/share/tools/web_config/js/colorutils.js @@ -339,21 +339,21 @@ var ayuTheme = { // ayu Light var color_scheme_ayu_light = { name: 'ayu Light', - url: 'https://github.com/edouard-lopez/ayu-theme.fish', + url: 'https://github.com/dempfi/ayu', } ayuTheme.apply(ayuTheme.ayu_light, color_scheme_ayu_light) // ayu Dark var color_scheme_ayu_dark = { name: 'ayu Dark', - url: 'https://github.com/edouard-lopez/ayu-theme.fish', + url: 'https://github.com/dempfi/ayu', } ayuTheme.apply(ayuTheme.ayu_dark, color_scheme_ayu_dark) // ayu Mirage var color_scheme_ayu_mirage = { name: 'ayu Mirage', - url: 'https://github.com/edouard-lopez/ayu-theme.fish', + url: 'https://github.com/dempfi/ayu', } ayuTheme.apply(ayuTheme.ayu_mirage, color_scheme_ayu_mirage) From 0507b046effba5849dc26965c76fadf81c29dd3c Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Mon, 4 Jan 2021 17:24:36 +0100 Subject: [PATCH 059/105] completions/pkg: Only exit for Solaris, not everything-but-FreeBSD In e8b67050679da41128fd8024420cfb9536acf78d this was made to exit if not on FreeBSD because Solaris has a tool called "pkg" that apparently "isn't worth supporting". Since at least DragonflyBSD also uses FreeBSD's pkg thing, let's turn that check around. --- share/completions/pkg.fish | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/share/completions/pkg.fish b/share/completions/pkg.fish index e2e03d0a5..87ff822df 100644 --- a/share/completions/pkg.fish +++ b/share/completions/pkg.fish @@ -1,6 +1,8 @@ # Completions for pkgng package manager -if uname | not string match -q FreeBSD +# Solaris has a thing called "pkg", it works quite differently, +# and spews errors when called like this. +if uname | string match -q SunOS exit end From cb9029944e50f218bd2c6f09ed62f977ab5bbc91 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Mon, 4 Jan 2021 21:51:13 +0100 Subject: [PATCH 060/105] docs: Remove margin entirely on small screens This removes the margin with the background gradient and such completely once the screen falls under 700px. In those cases we really don't want to waste space, and having just a weird blue bit above the docs looks weirder than not having anything. --- doc_src/python_docs_theme/static/pydoctheme.css | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/doc_src/python_docs_theme/static/pydoctheme.css b/doc_src/python_docs_theme/static/pydoctheme.css index 6cdeb8240..a37dd40d2 100644 --- a/doc_src/python_docs_theme/static/pydoctheme.css +++ b/doc_src/python_docs_theme/static/pydoctheme.css @@ -59,7 +59,8 @@ div.sphinxsidebar { float: none; } -/* On screens that are less than 700px wide remove the sidebar */ +/* On screens that are less than 700px wide remove anything non-essential + - the sidebar, the gradient background, ... */ @media screen and (max-width: 700px) { div.sphinxsidebar { width: 100%; @@ -68,6 +69,13 @@ div.sphinxsidebar { } div.content {margin-left: 0;} div.bodywrapper { margin: 0; } + div#fmain { + border-radius: 0px; + margin: 0; + -moz-box-shadow: 0; + -webkit-box-shadow: 0; + box-shadow: 0; + } } div.sphinxsidebar h3, div.sphinxsidebar h4 { From c1ef9676cb186bc33a2cd9dda1cb5f0e5ea8b748 Mon Sep 17 00:00:00 2001 From: Ben Woods Date: Mon, 4 Jan 2021 22:21:18 +0800 Subject: [PATCH 061/105] completions/pkg: Add support for "pkg check" sub-command --- share/completions/pkg.fish | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/share/completions/pkg.fish b/share/completions/pkg.fish index 87ff822df..db7576904 100644 --- a/share/completions/pkg.fish +++ b/share/completions/pkg.fish @@ -96,6 +96,26 @@ complete -c pkg -n '__fish_pkg_is add autoremove clean delete remove install upd complete -c pkg -n '__fish_pkg_is autoremove clean delete remove install upgrade' -s n -l dry-run -d "Do not make changes" complete -c pkg -n '__fish_pkg_is autoremove clean delete remove install' -s y -l yes -d "Assume yes when asked for confirmation" +# check +set -l has_check_opt '__fish_contains_opt -s B shlibs -s d dependencies -s s checksums -s r recompute' +set -l has_all_opt '__fish_contains_opt -s a all' +complete -c pkg -n "__fish_pkg_is check" -f +complete -c pkg -n "__fish_pkg_is check; and not $has_check_opt" -xa "-B -d -s -r" +complete -c pkg -n "__fish_pkg_is check; and not $has_check_opt" -s B -l shlibs -d "Regenerate library dependency metadata" +complete -c pkg -n "__fish_pkg_is check; and not $has_check_opt" -s d -l dependencies -d "Check for and install missing dependencies" +complete -c pkg -n "__fish_pkg_is check; and not $has_check_opt" -s r -l recompute -d "Recalculate and set the checksums of installed packages" +complete -c pkg -n "__fish_pkg_is check; and not $has_check_opt" -s s -l checksums -d "Detect installed packages with invalid checksums" +complete -c pkg -n "__fish_pkg_is check; and $has_check_opt" -s n -l dry-run -d "Do not make changes" +complete -c pkg -n "__fish_pkg_is check; and $has_check_opt" -s q -l quiet -d "Force quiet output" +complete -c pkg -n "__fish_pkg_is check; and $has_check_opt" -s v -l verbose -d "Provide verbose output" +complete -c pkg -n "__fish_pkg_is check; and $has_check_opt" -s y -l yes -d "Assume yes when asked for confirmation" +complete -c pkg -n "__fish_pkg_is check; and $has_check_opt; and not $has_all_opt" -xa '(pkg query "%n")' +complete -c pkg -n "__fish_pkg_is check; and $has_check_opt; and not $has_all_opt" -s a -l all -d "Process all packages" +complete -c pkg -n "__fish_pkg_is check; and $has_check_opt; and not $has_all_opt" -s C -l case-sensitive -d "Case sensitive packages" +complete -c pkg -n "__fish_pkg_is check; and $has_check_opt; and not $has_all_opt" -s g -l glob -d "Treat the package name as shell glob" +complete -c pkg -n "__fish_pkg_is check; and $has_check_opt; and not $has_all_opt" -s i -l case-insensitive -d "Case insensitive packages" +complete -c pkg -n "__fish_pkg_is check; and $has_check_opt; and not $has_all_opt" -s x -l regex -d "Treat the package name as regular expression" + # clean complete -c pkg -n '__fish_pkg_is clean' -s a -l all -d "Delete all cached packages" From 7669e8e4977103da9225403824b7f7f03f767779 Mon Sep 17 00:00:00 2001 From: Mahmoud Al-Qudsi Date: Tue, 5 Jan 2021 15:40:09 -0600 Subject: [PATCH 062/105] Add concept of edit groups This allows for multiple edits to be undone/redone in one go, as if they were one edit. Useful when a function is editing the commandline buffer via scripted changes or via a keybinding so the internal changes to the buffer can be abstracted away. (Having extreme difficulty getting pexpect to play nice with the concept of undo/redo...) --- share/functions/fish_clipboard_paste.fish | 3 + src/builtin_commandline.cpp | 17 +++- src/input.cpp | 2 + src/input_common.h | 2 + src/reader.cpp | 106 ++++++++++++++++++---- src/reader.h | 20 ++++ 6 files changed, 129 insertions(+), 21 deletions(-) diff --git a/share/functions/fish_clipboard_paste.fish b/share/functions/fish_clipboard_paste.fish index 9e766d667..2dcc823db 100644 --- a/share/functions/fish_clipboard_paste.fish +++ b/share/functions/fish_clipboard_paste.fish @@ -38,7 +38,10 @@ function fish_clipboard_paste # so we don't trigger ignoring history. set data[1] (string trim -l -- $data[1]) end + if test -n "$data" + begin-undo-group commandline -i -- $data + end-undo-group end end diff --git a/src/builtin_commandline.cpp b/src/builtin_commandline.cpp index d53c54a63..da4c4afdc 100644 --- a/src/builtin_commandline.cpp +++ b/src/builtin_commandline.cpp @@ -36,6 +36,9 @@ enum { APPEND_MODE // insert at end of current token/command/buffer }; +/// Handle a single readline_cmd_t command out-of-band. +void reader_handle_command(readline_cmd_t cmd); + /// Replace/append/insert the selection with/at/after the specified string. /// /// \param begin beginning of selection @@ -302,8 +305,18 @@ maybe_t builtin_commandline(parser_t &parser, io_streams_t &streams, wchar_ if (mc == rl::repaint_mode || mc == rl::force_repaint || mc == rl::repaint) { if (ld.is_repaint) continue; } - // Inserts the readline function at the back of the queue. - reader_queue_ch(*mc); + + // HACK: Execute these right here and now so they can affect any insertions/changes + // made via bindings. The correct solution is to change all `commandline` + // insert/replace operations into readline functions with associated data, so that + // all queued `commandline` operations - including buffer modifications - are + // executed in order + if (mc == rl::begin_undo_group || mc == rl::end_undo_group) { + reader_handle_command(*mc); + } else { + // Inserts the readline function at the back of the queue. + reader_queue_ch(*mc); + } } else { streams.err.append_format(_(L"%ls: Unknown input function '%ls'"), cmd, argv[i]); builtin_print_error_trailer(parser, streams.err, cmd); diff --git a/src/input.cpp b/src/input.cpp index cd3c22eba..ffaf6558f 100644 --- a/src/input.cpp +++ b/src/input.cpp @@ -157,6 +157,8 @@ static const input_function_metadata_t input_function_metadata[] = { {readline_cmd_t::cancel, L"cancel"}, {readline_cmd_t::undo, L"undo"}, {readline_cmd_t::redo, L"redo"}, + {readline_cmd_t::begin_undo_group, L"begin-undo-group"}, + {readline_cmd_t::end_undo_group, L"end-undo-group"}, }; static_assert(sizeof(input_function_metadata) / sizeof(input_function_metadata[0]) == diff --git a/src/input_common.h b/src/input_common.h index 09fe9dba6..35cf30387 100644 --- a/src/input_common.h +++ b/src/input_common.h @@ -80,6 +80,8 @@ enum class readline_cmd_t { cancel, undo, redo, + begin_undo_group, + end_undo_group, repeat_jump, // NOTE: This one has to be last. reverse_repeat_jump diff --git a/src/reader.cpp b/src/reader.cpp index 42e907d9a..2c4b8e5b3 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -200,7 +200,7 @@ static bool want_to_coalesce_insertion_of(const editable_line_t &el, const wcstr // Only consolidate single character inserts. if (str.size() != 1) return false; // Make an undo group after every space. - if (str.at(0) == L' ') return false; + if (str.at(0) == L' ' && !el.undo_history.try_coalesce) return false; assert(!el.undo_history.edits.empty()); const edit_t &last_edit = el.undo_history.edits.back(); // Don't add to the last edit if it deleted something. @@ -211,19 +211,35 @@ static bool want_to_coalesce_insertion_of(const editable_line_t &el, const wcstr } bool editable_line_t::undo() { - if (undo_history.edits_applied == 0) return false; // nothing to undo - const edit_t &edit = undo_history.edits.at(undo_history.edits_applied - 1); - undo_history.edits_applied--; - edit_t inverse = edit_t(edit.offset, edit.replacement.size(), L""); - inverse.replacement = edit.old; - size_t old_position = edit.cursor_position_before_edit; - apply_edit(&text_, inverse); - set_position(old_position); + bool did_undo = false; + maybe_t last_group_id{-1}; + while (undo_history.edits_applied != 0) { + const edit_t &edit = undo_history.edits.at(undo_history.edits_applied - 1); + if (did_undo && (!edit.group_id.has_value() || edit.group_id != last_group_id)) { + // We've restored all the edits in this logical undo group + break; + } + last_group_id = edit.group_id; + undo_history.edits_applied--; + edit_t inverse = edit_t(edit.offset, edit.replacement.size(), L""); + inverse.replacement = edit.old; + size_t old_position = edit.cursor_position_before_edit; + apply_edit(&text_, inverse); + set_position(old_position); + did_undo = true; + } + + end_edit_group(); undo_history.may_coalesce = false; - return true; + return did_undo; } void editable_line_t::push_edit(edit_t &&edit) { + // Assign a new group id or propagate the old one if we're in a logical grouping of edits + if (edit_group_level_ != -1) { + edit.group_id = edit_group_id_; + } + bool edit_does_nothing = edit.length == 0 && edit.replacement.empty(); if (edit_does_nothing) return; if (undo_history.edits_applied != undo_history.edits.size()) { @@ -251,13 +267,48 @@ void editable_line_t::insert_coalesce(const wcstring &str) { } bool editable_line_t::redo() { - if (undo_history.edits_applied >= undo_history.edits.size()) return false; // nothing to redo - const edit_t &edit = undo_history.edits.at(undo_history.edits_applied); - undo_history.edits_applied++; - apply_edit(&text_, edit); - set_position(cursor_position_after_edit(edit)); - undo_history.may_coalesce = false; // Make a new undo group here. - return true; + bool did_redo = false; + + maybe_t last_group_id{-1}; + while (undo_history.edits_applied < undo_history.edits.size()) { + const edit_t &edit = undo_history.edits.at(undo_history.edits_applied); + if (did_redo && (!edit.group_id.has_value() || edit.group_id != last_group_id)) { + // We've restored all the edits in this logical undo group + break; + } + last_group_id = edit.group_id; + undo_history.edits_applied++; + apply_edit(&text_, edit); + set_position(cursor_position_after_edit(edit)); + did_redo = true; + } + + end_edit_group(); + return did_redo; +} + +void editable_line_t::begin_edit_group() { + if (++edit_group_level_ == 0) { + // Indicate that the next change must trigger the creation of a new history item + undo_history.may_coalesce = false; + // Indicate that future changes should be coalesced into the same edit if possible. + undo_history.try_coalesce = true; + // Assign a logical edit group id to future edits in this group + edit_group_id_ += 1; + } +} + +void editable_line_t::end_edit_group() { + if (edit_group_level_ == -1) { + // Clamp the minimum value to -1 to prevent unbalanced end_edit_group() calls from breaking + // everything. + return; + } + + if (--edit_group_level_ == -1) { + undo_history.try_coalesce = false; + undo_history.may_coalesce = false; + } } namespace { @@ -1415,7 +1466,7 @@ void reader_data_t::insert_string(editable_line_t *el, const wcstring &str) { assert(el->undo_history.may_coalesce); } else { el->push_edit(edit_t(el->position(), 0, str)); - el->undo_history.may_coalesce = (str.size() == 1); + el->undo_history.may_coalesce = el->undo_history.try_coalesce || (str.size() == 1); } if (el == &command_line) suppress_autosuggestion = false; @@ -3710,7 +3761,17 @@ void reader_data_t::handle_readline_command(readline_cmd_t c, readline_loop_stat } break; } - // Some commands should have been handled internally by inputter_t::readch(). + case rl::begin_undo_group: { + editable_line_t *el = active_edit_line(); + el->begin_edit_group(); + break; + } + case rl::end_undo_group: { + editable_line_t *el = active_edit_line(); + el->end_edit_group(); + break; + } + // Some commands should have been handled internally by inputter_t::readch(). case rl::self_insert: case rl::self_insert_notfirst: case rl::func_or: @@ -3981,6 +4042,13 @@ void reader_schedule_prompt_repaint() { } } +void reader_handle_command(readline_cmd_t cmd) { + if (reader_data_t *data = current_data_or_null()) { + readline_loop_state_t rls{}; + data->handle_readline_command(cmd, rls); + } +} + void reader_queue_ch(const char_event_t &ch) { if (reader_data_t *data = current_data_or_null()) { data->inputter.queue_ch(ch); diff --git a/src/reader.h b/src/reader.h index 18fbadd35..7acb76c3a 100644 --- a/src/reader.h +++ b/src/reader.h @@ -31,6 +31,10 @@ struct edit_t { /// The strings that are removed and added by this edit, respectively. wcstring old, replacement; + /// edit_t is only for contiguous changes, so to restore a group of arbitrary changes to the + /// command line we need to have a group id as forcibly coalescing changes is not enough. + maybe_t group_id; + explicit edit_t(size_t offset, size_t length, wcstring replacement) : offset(offset), length(length), replacement(std::move(replacement)) {} @@ -61,6 +65,11 @@ struct undo_history_t { /// last one. bool may_coalesce = false; + /// Whether to be more aggressive in coalescing edits. Ideally, it would be "force coalesce" + /// with guaranteed atomicity but as `edit_t` is strictly for contiguous changes, that guarantee + /// can't be made at this time. + bool try_coalesce = false; + /// Empty the history. void clear(); }; @@ -72,6 +81,12 @@ class editable_line_t { /// The current position of the cursor in the command line. size_t position_ = 0; + /// The nesting level for atomic edits, so that recursive invocations of start_edit_group() + /// are not ended by one end_edit_group() call. + int32_t edit_group_level_ = -1; + /// Monotonically increasing edit group, ignored when edit_group_level_ is -1. Allowed to wrap. + uint32_t edit_group_id_ = -1; + public: undo_history_t undo_history; @@ -110,6 +125,11 @@ class editable_line_t { /// Redo the most recent undo. Returns true on success. bool redo(); + + /// Start a logical grouping of command line edits that should be undone/redone together. + void begin_edit_group(); + /// End a logical grouping of command line edits that should be undone/redone together. + void end_edit_group(); }; /// Read commands from \c fd until encountering EOF. From b5523dbd64eb0fe80e2c21995cac91e76d74c827 Mon Sep 17 00:00:00 2001 From: Mahmoud Al-Qudsi Date: Tue, 5 Jan 2021 17:26:48 -0600 Subject: [PATCH 063/105] Restrict `pkg` completions to BSD 0507b04 loosened the FreeBSD-only restriction on `pkg` completions to !SunOS in order to support DragonFlyBSD. This is overly broad and can still cause the script to be loaded on systems that we can't realistically expect to have `pkg` be the FreeBSD pkgng package manager (especially since `pkg` is a much more generic term when compared to the likes of `dnf`, `yum`, `deb`, and `apt`). This patch changes `pkg` + BSD to be the minimum requirements for considering a system to be using pkgng. --- share/completions/pkg.fish | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/share/completions/pkg.fish b/share/completions/pkg.fish index db7576904..7bdca87f6 100644 --- a/share/completions/pkg.fish +++ b/share/completions/pkg.fish @@ -1,8 +1,14 @@ # Completions for pkgng package manager -# Solaris has a thing called "pkg", it works quite differently, -# and spews errors when called like this. -if uname | string match -q SunOS +# Solaris has a thing called "pkg"; it works quite differently and spews errors when called here. +# There are multiple SunOS-derived distributions and not all of them have `SunOS` in their name (and +# some of them also use pkgsrc and have a `pkg`). +# +# Additionally, this particular script is intended to complete the pkgng "Next Generation" package +# manager initially developed for FreeBSD though now available on a few other BSDs. From here on +# out, maintainers can assume we are specifically talking about the (Free)BSD `pkg` command being +# executed on a BSD system, rather than just work with "not SunOS". +if ! uname | string match -irq bsd exit end From aaaf74cd5b228630996698228ed41858cc5ef3ee Mon Sep 17 00:00:00 2001 From: Mahmoud Al-Qudsi Date: Wed, 6 Jan 2021 16:45:18 -0600 Subject: [PATCH 064/105] fixup! Add concept of edit groups Correctly call begin/end-undo-group in fish_clipboard_paste --- share/functions/fish_clipboard_paste.fish | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/share/functions/fish_clipboard_paste.fish b/share/functions/fish_clipboard_paste.fish index 2dcc823db..bb670fcb9 100644 --- a/share/functions/fish_clipboard_paste.fish +++ b/share/functions/fish_clipboard_paste.fish @@ -40,8 +40,8 @@ function fish_clipboard_paste end if test -n "$data" - begin-undo-group + commandline -f begin-undo-group commandline -i -- $data - end-undo-group + commandline -f end-undo-group end end From a0764ef3d2b9dcf6933a6982ab2d64576db74427 Mon Sep 17 00:00:00 2001 From: David Adam Date: Thu, 7 Jan 2021 15:44:01 +0800 Subject: [PATCH 065/105] docs: note limits on parameter expansion from #7226 introduced in 594a6a3 --- doc_src/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc_src/index.rst b/doc_src/index.rst index 23d38b3e3..7c582cf6e 100644 --- a/doc_src/index.rst +++ b/doc_src/index.rst @@ -490,6 +490,7 @@ When fish is given a commandline, it expands the parameters before sending them - :ref:`Brace expansion `, to write lists with common pre- or suffixes in a shorter way - :ref:`Tilde expansion `, to turn the ``~`` at the beginning of paths into the path to the home directory +Parameter expansion is limited to 524288 items. .. _expand-wildcard: From fb873f2e980b141be216949f30f55e37e94d37e8 Mon Sep 17 00:00:00 2001 From: David Adam Date: Thu, 7 Jan 2021 22:17:04 +0800 Subject: [PATCH 066/105] CHANGELOG: work on 3.2.0 --- CHANGELOG.rst | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 6e068d68c..e5822b1d8 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -32,14 +32,14 @@ Notable improvements and fixes 1 = 2 and echo true or echo false ^ -- The documentation (:issue:`6500`, :issue:`7371`) and ``fish_config`` (:issue:`7523`) received a new theme, matching the design on fishshell.com. +- The documentation (:issue:`6500`, :issue:`7371`) and Web-based configuration (:issue:`7523`) received a new theme, matching the design on fishshell.com. - ``fish --no-execute`` will no longer complain about unknown commands or non-matching wildcards, as these could be defined differently at runtime (especially for functions). This makes it usable as a static syntax checker (:issue:`977`). - ``type`` is now a builtin and therefore much faster (:issue:`7342`). -- ``string match --regex`` now imports named PCRE2 capture groups as fish variables (:issue:`7459`). Note: Because of this, it can no longer be wrapped in a function and the name has been added as a reserved word. +- ``string match --regex`` now imports named PCRE2 capture groups as fish variables (:issue:`7459`). To support this functionality, ``string`` is now a reserved word and can no longer be wrapped in a function. - Globs and other expansions are limited to 512k results (:issue:`7226`). Because operating systems limit arguments to ARG_MAX, larger values are unlikely to work anyway, and this helps to avoid hangs. -- fish will now always attempt to become process group leader in interactive mode (:issue:`7060`). This helps avoid hangs in certain circumstances, and allows tmux' cwd-introspection hack to work (:issue:`5699`). +- fish will now always attempt to become process group leader in interactive mode (:issue:`7060`). This helps avoid hangs in certain circumstances, and allows tmux's current directory introspection to work (:issue:`5699`). Syntax changes and new commands ------------------------------- @@ -82,7 +82,7 @@ Scripting improvements file descriptor. This allows better error recovery and is more in line with other shells' behaviour (:issue:`7038`). - ``jobs --quiet PID`` no longer prints "no suitable job" if the job for PID does not exist (eg because it has finished) (:issue:`6809`). -- All builtins that query if something exists now take ``--query``. ``--quiet`` is deprecated for ``command``, ``jobs`` and ``type`` (:issue:`7276`). +- ``command``, ``jobs`` and ``type`` builtins support ``--query`` as the long form of ``-q``, matching other builtins. The long form ``--quiet`` is deprecated (:issue:`7276`). - ``argparse`` no longer requires a short flag letter for long-only options (:issue:`7585`) and only prints a backtrace with invalid options to argparse itself (:issue:`6703`). - ``complete`` takes the first argument as the name of the command if the ``--command``/``-c`` option is not used (``complete git`` is treated like ``complete --command git``), and can show the loaded completions for specific commands with ``complete COMMANDNAME`` (:issue:`7321`). - ``set_color -b`` (without an argument) no longer prints an error message, matching other invalid invocations of this command (:issue:`7154`). @@ -96,7 +96,7 @@ Scripting improvements - ``test -t``, for testing whether file descriptors are connected to a terminal, works for file descriptors 0, 1, and 2 (:issue:`4766`). It can still return incorrect results in other cases (:issue:`1228`). - Trying to execute scripts with Windows line endings (CRLF) produces a sensible error (:issue:`2783`). - An ``alias`` that delegates to a command with the same name no longer triggers an error about recursive completion (:issue:`7389`). -- ``math`` now has a ``--base`` option to output the result in hexadecimal or octal (:issue:`7496`) and some more specific errors (:issue:`7508`). +- ``math`` now has a ``--base`` option to output the result in hexadecimal or octal (:issue:`7496`) and produces more specific error messages (:issue:`7508`). - ``math`` learned bitwise functions ``bitand``, ``bitor`` and ``bitxor``, used like ``math "bitand(0xFE, 5)"`` (:issue:`7281`). - ``math`` learned tau for those wishing to cut down on typing "2 * pi". - ``string`` subcommands now quit early when used with ``--quiet`` (:issue:`7495`). @@ -130,7 +130,7 @@ Interactive improvements revealed. - The output of ``time`` is now properly aligned in all cases (:issue:`6726`). - The ``pwd`` command supports the long options ``--logical`` and ``--physical``, matching other implementations (:issue:`6787`). -- The command-not-found handling has been simplified. When it can't find a command, fish now just executes a function called ``fish_command_not_found`` instead of firing an event, making it easier to replace and reason about. Shims for backwards-compatibility have been added (:issue:`7293`). +- The command-not-found handling has been simplified. When it can't find a command, fish now just executes a function called ``fish_command_not_found`` instead of firing an event, making it easier to replace and reason about. Previously-defined ``__fish_command_not_found_handler`` functions with an appropriate event listener will still work (:issue:`7293`). - Control-C no longer occasionally prints an "unknown command" error (:issue:`7145`) or overwrites multiline prompts (:issue:`3537`). - Autocompletions work properly after Control-C to cancel the commmand line (:issue:`6937`). - History search is now case-insensitive unless the search string contains an uppercase character (:issue:`7273`). @@ -141,7 +141,7 @@ Interactive improvements - ``functions -t`` works like the long option ``--handlers-type``, as documented, instead of producing an error (:issue:`6985`). - History search now flashes when it found no more results (:issue:`7362`) - fish's debugging can now also be enabled via $FISH_DEBUG and $FISH_DEBUG_OUTPUT from the outside. This helps with debugging when no commandline options can be passed, like when fish is called in a shebang (:issue:`7359`). -- fish now creates $XDG_RUNTIME_DIR if it does not exist (:issue:`7335`). +- fish now creates the path in the environment variable ``XDG_RUNTIME_DIR`` if it does not exist, before using it for runtime data storage (:issue:`7335`). - ``set_color --print-colors`` now also respects the bold, dim, underline, reverse, italic and background modifiers, to better show their effect (:issue:`7314`). - The fish Web configuration tool (``fish_config``) shows prompts correctly on Termux for Android (:issue:`7298`) and detects Windows Services for Linux 2 properly (:issue:`7027`). - ``funcsave`` has a new ``--directory`` option to specify the location of the saved function (:issue:`7041`). @@ -149,7 +149,7 @@ Interactive improvements - Resuming a piped job by its number, like ``fg %1``, works correctly (:issue:`7406`). Resumed jobs show the correct title in the terminal emulator (:issue:`7444`). - Commands run from key bindings now use the same TTY modes as normal commands (:issue:`7483`). - Autosuggestions from history are now case-sensitive, and tab completions are "smartcase": they offer case-insensitive matches if the input string is lowercase (:issue:`3978`). -- ``$status`` from completion scripts is no longer visible outside, like in the prompt - this prevents status display in the prompt from being overwritten (:issue:`7555`). +- ``$status`` from completion scripts is no longer passed outside the completion, which keeps the status display in the prompt as the last command's status (:issue:`7555`). - Updated localisations for pt_BR (:issue:`7480`). - ``fish_trace`` output now starts with ``->`` like ``fish --profile``'s, making the depth more visible (:issue:`7538`). - Resizing the terminal window no longer produces a corrupted prompt (:issue:`6532`). @@ -180,7 +180,7 @@ New or improved bindings - ``__fish_toggle_comment_commandline`` (Alt-#) now uncomments and presents the last comment from history if the commandline is empty (:issue:`7137`). - ``__fish_whatis_current_token`` (Alt-W) prints descriptions for functions and builtins (:issue:`7191`). -- The definition of "word" and "bigword" for movements was refined, fixing e.g. vi mode's behavior with ``e`` on the second-to-last char, and bigword's behavior with single-char words and non-blank non-graphic characters (:issue:`7353`, :issue:`7354`, :issue:`4025`, :issue:`7328`, :issue:`7325`) +- The definition of "word" and "bigword" for movements was refined, fixing (eg) vi mode's behavior with ``e`` on the second-to-last char, and bigword's behavior with single-char words and non-blank non-graphic characters (:issue:`7353`, :issue:`7354`, :issue:`4025`, :issue:`7328`, :issue:`7325`) - fish's clipboard bindings now also support WSL via powershell and clip.exe (:issue:`7455`). Improved prompts @@ -202,9 +202,8 @@ Improved terminal support - A new variable, ``$fish_vi_force_cursor``, can be set to force ``fish_vi_cursor`` to attempt changing the cursor shape in vi mode, regardless of terminal (:issue:`6968`). The ``fish_vi_cursor`` option ``--force-iterm`` has been deprecated. -- ``diff`` will now colourise output, if supported (:issue:`7308`). -- Autosuggestions now show up also when the cursor passes the right - prompt (:issue:`6948`) or wraps to the next line (:issue:`7213`). +- ``diff`` will now colourize output, if supported (:issue:`7308`). +- Autosuggestions appear when the cursor passes the right prompt (:issue:`6948`) or wraps to the next line (:issue:`7213`). - The cursor shape in Vi mode changes properly in Windows Terminal (:issue:`6999`). - The spurious warning about terminal size in small terminals has been removed (:issue:`6980`). - Dynamic titles are now enabled in Alacritty (:issue:`7073`). @@ -283,8 +282,8 @@ Completions Deprecations and removed features --------------------------------- -- fish no longer attempts to modify the terminal size via `TIOCSWINSZ` (:issue:`6994`). -- The `fish_color_match` variable is no longer used. (Previously this controlled the color of matching quotes and parens when using `read`). +- fish no longer attempts to modify the terminal size via ``TIOCSWINSZ`` (:issue:`6994`). +- The ``fish_color_match`` variable is no longer used. (Previously this controlled the color of matching quotes and parens when using ``read``). - fish 3.2.0 will be the last release in which the redirection to standard error with the ``^`` character is enabled. The ``stderr-nocaret`` feature flag will be changed to "on" in future releases. For distributors and developers @@ -303,8 +302,8 @@ For distributors and developers codesigning is enabled (:issue:`6952`). - Running the full interactive test suite now requires Python 3.5+ and the pexpect package (:issue:`6825`); the expect package is no longer required. - Support for Python 2 in fish's tools (``fish_config`` and the manual page completion generator) is no longer guaranteed. Please use Python 3.5 or later (:issue:`6537`). -- The web-based configuration tool no longer requires Python's distutils (:issue:`7514`) or the deprecated cgi module (:issue:`7600`). -- fish 3.2 is the last release to support Red Hat Enterprise Linux & CentOS version 7. +- The Web-based configuration tool is compatible with Python 3.10 (:issue:`7600`) and no longer requires Python's distutils package (:issue:`7514`). +- fish 3.2 is the last release to support Red Hat Enterprise Linux & CentOS version 6. -------------- From 7a53c40fd43ed362debe22393c3d506bb6ad60be Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Thu, 7 Jan 2021 16:51:20 +0100 Subject: [PATCH 067/105] Allow to run individual interactive tests by setting FISH_PEXPECT_FILES This command builds all test dependencies and runs the bind.py test: FISH_PEXPECT_FILES=../tests/pexpects/bind.py ninja test_interactive --- tests/interactive.fish | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/interactive.fish b/tests/interactive.fish index 4f5d39041..6a1228453 100644 --- a/tests/interactive.fish +++ b/tests/interactive.fish @@ -20,6 +20,8 @@ set -e ITERM_PROFILE # Test files specified on commandline, or all pexpect files. if set -q argv[1] set pexpect_files_to_test pexpects/$argv.py +else if set -q FISH_PEXPECT_FILES + set pexpect_files_to_test (string replace -r '^.*/(?=pexpects/)' '' -- $FISH_PEXPECT_FILES) else set pexpect_files_to_test pexpects/*.py end From 0729c2be4c1172fa4334926c6e60fdec2a867633 Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Thu, 7 Jan 2021 16:53:15 +0100 Subject: [PATCH 068/105] Re-add completions for source and ., to prefer *.fish files This is mildly useful when activating virtualenvs. We had remove these files earlier, but since there are no more false negatives from __fish_complete_suffix it seems safe to re-add them. --- share/completions/..fish | 1 + share/completions/source.fish | 2 ++ 2 files changed, 3 insertions(+) create mode 100644 share/completions/..fish create mode 100644 share/completions/source.fish diff --git a/share/completions/..fish b/share/completions/..fish new file mode 100644 index 000000000..8dc864d1a --- /dev/null +++ b/share/completions/..fish @@ -0,0 +1 @@ +complete . -w source diff --git a/share/completions/source.fish b/share/completions/source.fish new file mode 100644 index 000000000..f80bd6b8f --- /dev/null +++ b/share/completions/source.fish @@ -0,0 +1,2 @@ +complete source -k -xa '(__fish_complete_suffix .fish)' +complete source -s h -l help -d 'Display help and exit' From 534bc66a435af7899fa4131021648be8749585f8 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Sun, 3 Jan 2021 16:42:30 -0800 Subject: [PATCH 069/105] Add a test for background procs in cmdsubs This adds a test to ensure that if a long running background process is launched from a command substitution, that process does not cause the cmdsub to hang. That could easily happen if we just wait for the pipe to close; this is verifying that we are also checking for the job to complete. --- tests/checks/cmdsub-limit.fish | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/tests/checks/cmdsub-limit.fish b/tests/checks/cmdsub-limit.fish index 7b0ee047c..bb286d94f 100644 --- a/tests/checks/cmdsub-limit.fish +++ b/tests/checks/cmdsub-limit.fish @@ -1,8 +1,25 @@ # RUN: %fish %s -# This tests various corner cases involving command substitution. Most -# importantly the limits on the amount of data we'll substitute. +# This tests various corner cases involving command substitution. +# Test cmdsubs which spawn background processes - see #7559. +# If this hangs, it means that fish keeps trying to read from the write +# end of the cmdsub pipe (which has escaped). + +# FIXME: we need to mark full job control for sleep to get its own pgid; +# otherwise $last_pid will return fish's pgid! It's always been so! +status job-control full + +echo (command sleep 1000000 & ; set -g sleep_pid $last_pid ; echo local) +# CHECK: local +echo $sleep_pid +# CHECK: {{[1-9]\d*}} +kill $sleep_pid ; echo $status +# CHECK: 0 + +status job-control interactive + +# Test limiting the amount of data we'll substitute. set fish_read_limit 512 function subme @@ -10,7 +27,7 @@ function subme echo $x end -# Command sub just under the limit should succeed +# Command sub just under the limit should succeed. set a (subme 511) set --show a #CHECK: $a: set in global scope, unexported, with 1 elements From fd08b660c0ba07c8eb1d966c4160f6b58ea8a210 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Sun, 3 Jan 2021 16:26:28 -0800 Subject: [PATCH 070/105] Add a poke function to fd_monitor In preparation for fixing #7559, add a function poke_item to fd_monitor. fd_monitor has a list of file descriptors, and invokes a callback when an fd becomes readable. With this change, we assign each item a unique ID and return it when the item is added; the ID may then be used to invoke the callback explicitly. The idea is that we can stop reading from the pipe associated with the cmdsub when the job is finished, even if the pipe is still open. --- src/fd_monitor.cpp | 97 +++++++++++++++++++++++++++++++++++++--------- src/fd_monitor.h | 57 +++++++++++++++++++++------ src/fish_tests.cpp | 48 +++++++++++++++++------ src/io.cpp | 6 +-- 4 files changed, 164 insertions(+), 44 deletions(-) diff --git a/src/fd_monitor.cpp b/src/fd_monitor.cpp index 00b7a717d..d85b86014 100644 --- a/src/fd_monitor.cpp +++ b/src/fd_monitor.cpp @@ -26,15 +26,16 @@ fd_monitor_t::fd_monitor_t() { // Add an item for ourselves. // We don't need to go through 'pending' because we have not yet launched the thread, and don't // want to yet. - auto callback = [this](const autoclose_fd_t &fd, bool timed_out) { + auto callback = [this](const autoclose_fd_t &fd, item_wake_reason_t reason) { ASSERT_IS_BACKGROUND_THREAD(); - assert(!timed_out && "Should not time out with kNoTimeout"); - (void)timed_out; + assert(reason == item_wake_reason_t::readable && + "Should not be poked, or time out with kNoTimeout"); + (void)reason; // Read some to take data off of the notifier. char buff[4096]; ssize_t amt = read(fd.fd(), buff, sizeof buff); if (amt > 0) { - this->has_pending_ = true; + this->has_pending_or_pokes_ = true; } else if (amt == 0) { this->terminate_ = true; } else { @@ -54,10 +55,28 @@ fd_monitor_t::~fd_monitor_t() { } } -void fd_monitor_t::add(fd_monitor_item_t &&item) { +fd_monitor_item_id_t fd_monitor_t::add(fd_monitor_item_t &&item) { assert(item.fd.valid() && "Invalid fd"); assert(item.timeout_usec != 0 && "Invalid timeout"); - bool start_thread = add_pending_get_start_thread(std::move(item)); + assert(item.item_id == 0 && "Item should not already have an ID"); + bool start_thread = false; + fd_monitor_item_id_t item_id{}; + { + // Lock around a local region. + auto data = data_.acquire(); + + // Assign an id and add the item to pending. + item_id = ++data->last_id; + item.item_id = item_id; + data->pending.push_back(std::move(item)); + + // Maybe plan to start the thread. + if (!data->running) { + FLOG(fd_monitor, "Thread starting"); + data->running = true; + start_thread = true; + } + } if (start_thread) { void *(*trampoline)(void *) = [](void *self) -> void * { static_cast(self)->run_in_background(); @@ -71,17 +90,24 @@ void fd_monitor_t::add(fd_monitor_item_t &&item) { // Tickle our notifier. char byte = 0; (void)write_loop(notify_write_fd_.fd(), &byte, 1); + return item_id; } -bool fd_monitor_t::add_pending_get_start_thread(fd_monitor_item_t &&item) { - auto data = data_.acquire(); - data->pending.push_back(std::move(item)); - if (!data->running) { - FLOG(fd_monitor, "Thread starting"); - data->running = true; - return true; +void fd_monitor_t::poke_item(fd_monitor_item_id_t item_id) { + assert(item_id > 0 && "Invalid item ID"); + bool needs_notifier_byte = false; + { + auto data = data_.acquire(); + needs_notifier_byte = data->pokelist.empty(); + // Insert it, sorted. + auto where = std::lower_bound(data->pokelist.begin(), data->pokelist.end(), item_id); + data->pokelist.insert(where, item_id); + } + if (needs_notifier_byte) { + // Tickle our notifier. + char byte = 0; + (void)write_loop(notify_write_fd_.fd(), &byte, 1); } - return false; } // Given a usec count, populate and return a timeval. @@ -108,15 +134,33 @@ bool fd_monitor_item_t::service_item(const fd_set *fds, const time_point_t &now) bool timed_out = !readable && usec_remaining(now) == 0; if (readable || timed_out) { last_time = now; - callback(fd, timed_out); + item_wake_reason_t reason = + readable ? item_wake_reason_t::readable : item_wake_reason_t::timeout; + callback(fd, reason); should_retain = fd.valid(); } return should_retain; } +bool fd_monitor_item_t::poke_item(const poke_list_t &pokelist) { + if (item_id == 0 || !std::binary_search(pokelist.begin(), pokelist.end(), item_id)) { + // Not pokeable or not in the pokelist. + return true; + } + callback(fd, item_wake_reason_t::poke); + return fd.valid(); +} + void fd_monitor_t::run_in_background() { ASSERT_IS_BACKGROUND_THREAD(); + poke_list_t pokelist; for (;;) { + // Poke any items that need it. + if (!pokelist.empty()) { + this->poke_in_background(std::move(pokelist)); + pokelist.clear(); + } + uint64_t timeout_usec = fd_monitor_item_t::kNoTimeout; int max_fd = -1; fd_set fds; @@ -158,7 +202,7 @@ void fd_monitor_t::run_in_background() { return remove; }; - // Service all items that are either readable or timed our, and remove any which say to do + // Service all items that are either readable or timed out, and remove any which say to do // so. now = std::chrono::steady_clock::now(); items_.erase(std::remove_if(items_.begin(), items_.end(), servicer), items_.end()); @@ -171,13 +215,19 @@ void fd_monitor_t::run_in_background() { // Maybe we got some new items. Check if our callback says so, or if this is the wait // lap, in which case we might want to commit to exiting. - if (has_pending_ || is_wait_lap) { + if (has_pending_or_pokes_ || is_wait_lap) { auto data = data_.acquire(); // Move from 'pending' to 'items'. items_.insert(items_.end(), std::make_move_iterator(data->pending.begin()), std::make_move_iterator(data->pending.end())); data->pending.clear(); - has_pending_ = false; + + // Grab any pokelist. + assert(pokelist.empty() && "pokelist should be empty or else we're dropping pokes"); + pokelist = std::move(data->pokelist); + data->pokelist.clear(); + + has_pending_or_pokes_ = false; if (is_wait_lap && items_.size() == 1) { // We had no items, waited a bit, and still have no items. We're going to shut down. @@ -191,3 +241,14 @@ void fd_monitor_t::run_in_background() { } } } + +void fd_monitor_t::poke_in_background(const poke_list_t &pokelist) { + ASSERT_IS_BACKGROUND_THREAD(); + auto poker = [&pokelist](fd_monitor_item_t &item) { + int fd = item.fd.fd(); + bool remove = !item.poke_item(pokelist); + if (remove) FLOG(fd_monitor, "Removing fd", fd); + return remove; + }; + items_.erase(std::remove_if(items_.begin(), items_.end(), poker), items_.end()); +} diff --git a/src/fd_monitor.h b/src/fd_monitor.h index 2c4fb0425..1127e102c 100644 --- a/src/fd_monitor.h +++ b/src/fd_monitor.h @@ -12,14 +12,24 @@ class fd_monitor_t; +/// Each item added to fd_monitor_t is assigned a unique ID, which is not recycled. +/// Items may have their callback triggered immediately by passing the ID. +/// Zero is a sentinel. +using fd_monitor_item_id_t = uint64_t; + +/// Reasons for waking an item. +enum class item_wake_reason_t { + readable, // the fd became readable + timeout, // the requested timeout was hit + poke, // the item was "poked" (woken up explicitly) +}; + /// An item containing an fd and callback, which can be monitored to watch when it becomes readable, /// and invoke the callback. struct fd_monitor_item_t { - friend class fd_monitor_t; - - /// The callback type for the item. - /// It will be invoked when either \p fd is readable, or if the timeout was hit. - using callback_t = std::function; + /// The callback type for the item. It is passed \p fd, and the reason for waking \p reason. + /// The callback may close \p fd, in which case the item is removed. + using callback_t = std::function; /// A sentinel value meaning no timeout. static constexpr uint64_t kNoTimeout = std::numeric_limits::max(); @@ -51,6 +61,9 @@ struct fd_monitor_item_t { // The last time we were called, or the initialization point. maybe_t last_time{}; + // The ID for this item. This is assigned by the fd monitor. + fd_monitor_item_id_t item_id{0}; + // \return the number of microseconds until the timeout should trigger, or kNoTimeout for none. // A 0 return means we are at or past the timeout. uint64_t usec_remaining(const time_point_t &now) const; @@ -58,6 +71,13 @@ struct fd_monitor_item_t { // Invoke this item's callback if its value is set in fd or has timed out. // \return true to retain the item, false to remove it. bool service_item(const fd_set *fds, const time_point_t &now); + + // Invoke this item's callback with a poke, if its ID is present in the (sorted) pokelist. + // \return true to retain the item, false to remove it. + using poke_list_t = std::vector; + bool poke_item(const poke_list_t &pokelist); + + friend class fd_monitor_t; }; /// A class which can monitor a set of fds, invoking a callback when any becomes readable, or when @@ -66,34 +86,47 @@ class fd_monitor_t { public: using item_list_t = std::vector; + // A "pokelist" is a sorted list of item IDs which need explicit wakeups. + using poke_list_t = std::vector; + fd_monitor_t(); ~fd_monitor_t(); - /// Add an item to monitor. - void add(fd_monitor_item_t &&item); + /// Add an item to monitor. \return the ID assigned to the item. + fd_monitor_item_id_t add(fd_monitor_item_t &&item); + + /// Mark that an item with a given ID needs to be explicitly woken up. + void poke_item(fd_monitor_item_id_t item_id); private: // The background thread runner. void run_in_background(); - // Add a pending item, marking the thread as running. - // \return true if we should start the thread. - bool add_pending_get_start_thread(fd_monitor_item_t &&item); + // Poke items in the pokelist, removing any items that close their FD. + // The pokelist is consumed after this. + // This is only called in the background thread. + void poke_in_background(const poke_list_t &pokelist); // The list of items to monitor. This is only accessed on the background thread. item_list_t items_{}; // Set to true by the background thread when our self-pipe becomes readable. - bool has_pending_{false}; + bool has_pending_or_pokes_{false}; // Latched to true by the background thread if our self-pipe is closed, which indicates we are // in the destructor and so should terminate. bool terminate_{false}; struct data_t { - /// Pending items. + /// Pending items. This is set under the lock, then the background thread grabs them. item_list_t pending{}; + /// List of IDs for items that need to be poked (explicitly woken up). + poke_list_t pokelist{}; + + /// The last ID assigned, or if none. + fd_monitor_item_id_t last_id{0}; + /// Whether the thread is running. bool running{false}; }; diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 379a475bc..2784e2b72 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -787,7 +787,9 @@ static void test_fd_monitor() { struct item_maker_t { std::atomic did_timeout{false}; std::atomic length_read{0}; + std::atomic pokes{0}; std::atomic total_calls{0}; + fd_monitor_item_id_t item_id{0}; bool always_exit{false}; fd_monitor_item_t item; autoclose_fd_t writer; @@ -795,15 +797,21 @@ static void test_fd_monitor() { explicit item_maker_t(uint64_t timeout_usec) { auto pipes = make_autoclose_pipes({}).acquire(); writer = std::move(pipes.write); - auto callback = [this](autoclose_fd_t &fd, bool timed_out) { + auto callback = [this](autoclose_fd_t &fd, item_wake_reason_t reason) { bool was_closed = false; - if (timed_out) { - this->did_timeout = true; - } else { - char buff[4096]; - ssize_t amt = read(fd.fd(), buff, sizeof buff); - length_read += amt; - was_closed = (amt == 0); + switch (reason) { + case item_wake_reason_t::timeout: + this->did_timeout = true; + break; + case item_wake_reason_t::poke: + this->pokes += 1; + break; + case item_wake_reason_t::readable: + char buff[4096]; + ssize_t amt = read(fd.fd(), buff, sizeof buff); + this->length_read += amt; + was_closed = (amt == 0); + break; } total_calls += 1; if (always_exit || was_closed) { @@ -840,45 +848,63 @@ static void test_fd_monitor() { // Item which should get 42 bytes, then get notified it is closed. item_maker_t item42_thenclose(16 * usec_per_msec); + // Item which gets one poke. + item_maker_t item_pokee(fd_monitor_item_t::kNoTimeout); + // Item which should be called back once. item_maker_t item_oneshot(16 * usec_per_msec); item_oneshot.always_exit = true; + { fd_monitor_t monitor; - for (auto item : {&item_never, &item_hugetimeout, &item0_timeout, &item42_timeout, - &item42_nottimeout, &item42_thenclose, &item_oneshot}) { - monitor.add(std::move(item->item)); + for (item_maker_t *item : + {&item_never, &item_hugetimeout, &item0_timeout, &item42_timeout, &item42_nottimeout, + &item42_thenclose, &item_pokee, &item_oneshot}) { + item->item_id = monitor.add(std::move(item->item)); } item42_timeout.write42(); item42_nottimeout.write42(); item42_thenclose.write42(); item42_thenclose.writer.close(); item_oneshot.write42(); + monitor.poke_item(item_pokee.item_id); std::this_thread::sleep_for(std::chrono::milliseconds(84)); } do_test(!item_never.did_timeout); do_test(item_never.length_read == 0); + do_test(item_never.pokes == 0); do_test(!item_hugetimeout.did_timeout); do_test(item_hugetimeout.length_read == 0); + do_test(item_hugetimeout.pokes == 0); do_test(item0_timeout.length_read == 0); do_test(item0_timeout.did_timeout); + do_test(item0_timeout.pokes == 0); do_test(item42_timeout.length_read == 42); do_test(item42_timeout.did_timeout); + do_test(item42_timeout.pokes == 0); do_test(item42_nottimeout.length_read == 42); do_test(!item42_nottimeout.did_timeout); + do_test(item42_nottimeout.pokes == 0); do_test(item42_thenclose.did_timeout == false); do_test(item42_thenclose.length_read == 42); do_test(item42_thenclose.total_calls == 2); + do_test(item42_thenclose.pokes == 0); do_test(!item_oneshot.did_timeout); do_test(item_oneshot.length_read == 42); do_test(item_oneshot.total_calls == 1); + do_test(item_oneshot.pokes == 0); + + do_test(!item_pokee.did_timeout); + do_test(item_pokee.length_read == 0); + do_test(item_pokee.total_calls == 1); + do_test(item_pokee.pokes == 1); } static void test_iothread() { diff --git a/src/io.cpp b/src/io.cpp index 1e5376146..ccb2a05be 100644 --- a/src/io.cpp +++ b/src/io.cpp @@ -124,14 +124,14 @@ void io_buffer_t::begin_filling(autoclose_fd_t fd) { fd_monitor_item_t item; item.fd = std::move(fd); item.timeout_usec = poll_usec; - item.callback = [this, promise](autoclose_fd_t &fd, bool timed_out) { + item.callback = [this, promise](autoclose_fd_t &fd, item_wake_reason_t reason) { ASSERT_IS_BACKGROUND_THREAD(); - // Only check the shutdown flag if we timed out. + // Only check the shutdown flag if we timed out or were poked. // It's important that if select() indicated we were readable, that we call select() again // allowing it to time out. Note the typical case is that the fd will be closed, in which // case select will return immediately. bool done = false; - if (!timed_out) { + if (reason == item_wake_reason_t::readable) { // select() reported us as readable; read a bit. scoped_lock locker(append_lock_); ssize_t ret = read_once(fd.fd()); From d5d09c993e1c569ec4d5ba4cb24ff706798382eb Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Wed, 6 Jan 2021 21:03:49 -0800 Subject: [PATCH 071/105] io_buffer_t to explicitly poke its item when closing io_buffer_t is used to buffer output from a command substitution, so we can split it into arguments. Typically io_buffer_t reads from its pipe until it gets EOF and then stops reading. However it may be that the cmdsub ends but EOF is not delivered because the stdout of the cmdsub escaped with a background process. Prior to this change we would wake up every 100 msec (select timeout) to check if the cmdsub is finished. However this 100 msec adds latency if a background process is launched from e.g. fish_prompt. Switch to the new poke() function. Now when the cmdsub is finished, it pokes its item, which explicitly wakes it up. This removes the extra latency. Fixes #7559 --- src/io.cpp | 5 ++++- src/io.h | 3 +++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/src/io.cpp b/src/io.cpp index ccb2a05be..9a8bcc0b0 100644 --- a/src/io.cpp +++ b/src/io.cpp @@ -151,13 +151,16 @@ void io_buffer_t::begin_filling(autoclose_fd_t fd) { promise->set_value(); } }; - fd_monitor().add(std::move(item)); + this->item_id_ = fd_monitor().add(std::move(item)); } void io_buffer_t::complete_background_fillthread() { + // Mark that our fillthread is done, then wake it up. ASSERT_IS_MAIN_THREAD(); assert(fillthread_running() && "Should have a fillthread"); + assert(this->item_id_ > 0 && "Should have a valid item ID"); shutdown_fillthread_ = true; + fd_monitor().poke_item(this->item_id_); // Wait for the fillthread to fulfill its promise, and then clear the future so we know we no // longer have one. diff --git a/src/io.h b/src/io.h index a1c9417cd..702954cf6 100644 --- a/src/io.h +++ b/src/io.h @@ -312,6 +312,9 @@ class io_buffer_t { /// running. The fillthread fulfills the corresponding promise when it exits. std::future fillthread_waiter_{}; + /// The item id of our background fillthread fd monitor item. + uint64_t item_id_{0}; + /// Lock for appending. std::mutex append_lock_{}; From 4faebf74e6320a331ef6fe0adc4a7b7ad2f0b80f Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Thu, 7 Jan 2021 12:06:21 -0800 Subject: [PATCH 072/105] Remove 100 msec timeout from io_buffer_t This removes the 100 msec timeout from io_buffer_t. We no longer need to periodically wake up to check if a command substitution is finished, because we get explicitly poked when that happens. --- src/io.cpp | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/src/io.cpp b/src/io.cpp index 9a8bcc0b0..815e1fe1a 100644 --- a/src/io.cpp +++ b/src/io.cpp @@ -94,7 +94,7 @@ void io_buffer_t::begin_filling(autoclose_fd_t fd) { // We want to fill buffer_ by reading from fd. fd is the read end of a pipe; the write end is // owned by another process, or something else writing in fish. // Pass fd to an fd_monitor. It will add fd to its select() loop, and give us a callback when - // the fd is readable, or when our timeout is hit. The usual path is that we will get called + // the fd is readable, or when our item is poked. The usual path is that we will get called // back, read a bit from the fd, and append it to the buffer. Eventually the write end of the // pipe will be closed - probably the other process exited - and fd will be widowed; read() will // then return 0 and we will stop reading. @@ -102,9 +102,10 @@ void io_buffer_t::begin_filling(autoclose_fd_t fd) { // e.g.: // cmd ( background & ; echo hi ) // Here the background process will inherit the write end of the pipe and hold onto it forever. - // In this case, we will hit the timeout on waiting for more data and notice that the shutdown - // flag is set (this indicates that the command substitution is done); in this case we will read - // until we get EAGAIN and then give up. + // In this case, when complete_background_fillthread() is called, the callback will be invoked + // with item_wake_reason_t::poke, and we will notice that the shutdown flag is set (this + // indicates that the command substitution is done); in this case we will read until we get + // EAGAIN and then give up. // Construct a promise that can go into our background thread. auto promise = std::make_shared>(); @@ -113,17 +114,10 @@ void io_buffer_t::begin_filling(autoclose_fd_t fd) { // Note this should only ever be called once. fillthread_waiter_ = promise->get_future(); - // 100 msec poll rate. Note that in most cases, the write end of the pipe will be closed so - // select() will return; the polling is important only for weird cases like a background process - // launched in a command substitution. - constexpr uint64_t usec_per_msec = 1000; - uint64_t poll_usec = 100 * usec_per_msec; - // Run our function to read until the receiver is closed. // It's OK to capture 'this' by value because 'this' waits for the promise in its dtor. fd_monitor_item_t item; item.fd = std::move(fd); - item.timeout_usec = poll_usec; item.callback = [this, promise](autoclose_fd_t &fd, item_wake_reason_t reason) { ASSERT_IS_BACKGROUND_THREAD(); // Only check the shutdown flag if we timed out or were poked. From 9f4255ed76683d6772f354c1fb818a1655e877a0 Mon Sep 17 00:00:00 2001 From: Johannes Altmanninger Date: Thu, 7 Jan 2021 23:52:20 +0100 Subject: [PATCH 073/105] Add simple pexpect test for undo This acts really strange, I haven't yet figured out why, but I guess it's a start. --- tests/pexpects/undo.py | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 tests/pexpects/undo.py diff --git a/tests/pexpects/undo.py b/tests/pexpects/undo.py new file mode 100644 index 000000000..faa335df2 --- /dev/null +++ b/tests/pexpects/undo.py @@ -0,0 +1,34 @@ +#!/usr/bin/env python3 +from pexpect_helper import SpawnedProc + +sp = SpawnedProc() +send, sendline, sleep, expect_prompt, expect_re, expect_str = ( + sp.send, + sp.sendline, + sp.sleep, + sp.expect_prompt, + sp.expect_re, + sp.expect_str, +) +expect_prompt() + +sendline("bind Undo undo; bind Redo redo") +expect_prompt() + +send("echo word") +expect_str("echo word") +expect_str("echo word") # Not sure why we get this twice. + +# FIXME why does this only undo one character? It undoes the entire word when run interactively. +send("Undo") +expect_str("echo wor") + +send("Undo") +expect_str("echo ") + +send("Redo") +expect_str("echo wor") + +# FIXME see above. +send("Redo") +expect_str("echo word") From 21f46181d9c343160eb9b01ca6038217b23f1539 Mon Sep 17 00:00:00 2001 From: David Adam Date: Fri, 8 Jan 2021 21:16:07 +0800 Subject: [PATCH 074/105] string match: reword the named capture group documentation --- CHANGELOG.rst | 2 +- doc_src/cmds/string-match.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index e5822b1d8..a62a8cd51 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -37,7 +37,7 @@ Notable improvements and fixes or non-matching wildcards, as these could be defined differently at runtime (especially for functions). This makes it usable as a static syntax checker (:issue:`977`). - ``type`` is now a builtin and therefore much faster (:issue:`7342`). -- ``string match --regex`` now imports named PCRE2 capture groups as fish variables (:issue:`7459`). To support this functionality, ``string`` is now a reserved word and can no longer be wrapped in a function. +- ``string match --regex`` now integrates named PCRE2 capture groups with fish variables, allowing variables to be set directly from ``string match`` (:issue:`7459`). To support this functionality, ``string`` is now a reserved word and can no longer be wrapped in a function. - Globs and other expansions are limited to 512k results (:issue:`7226`). Because operating systems limit arguments to ARG_MAX, larger values are unlikely to work anyway, and this helps to avoid hangs. - fish will now always attempt to become process group leader in interactive mode (:issue:`7060`). This helps avoid hangs in certain circumstances, and allows tmux's current directory introspection to work (:issue:`5699`). diff --git a/doc_src/cmds/string-match.rst b/doc_src/cmds/string-match.rst index b9a865123..1f05f4a6a 100644 --- a/doc_src/cmds/string-match.rst +++ b/doc_src/cmds/string-match.rst @@ -27,7 +27,7 @@ If ``--index`` or ``-n`` is given, each match is reported as a 1-based start pos If ``--regex`` or ``-r`` is given, PATTERN is interpreted as a Perl-compatible regular expression, which does not have to match the entire STRING. For a regular expression containing capturing groups, multiple items will be reported for each match, one for the entire match and one for each capturing group. With this, only the matching part of the STRING will be reported, unless ``--entire`` is given. -When matching via regular expressions, ``string match`` automatically imports all named capturing groups (``(?expression)``) as fish variables of the same name. It will create a variable in the default scope for each named capturing group, and set it to the value of the capturing group in the first matched argument. If a named capture group matched an empty string, the variable will be set to the empty string (like ``set var ""``). If it did not match, the variable will be set to nothing (like ``set var``). When ``--regex`` is used with ``--all``, this behavior changes. Each named variable will contain a list of matches, with the first match contained in the first element, the second match in the second, and so on. If the group was empty or did not match, the corresponding element will be an empty string. +When matching via regular expressions, ``string match`` automatically sets variables for all named capturing groups (``(?expression)``). It will create a variable with the name of the group, in the default scope, for each named capturing group, and set it to the value of the capturing group in the first matched argument. If a named capture group matched an empty string, the variable will be set to the empty string (like ``set var ""``). If it did not match, the variable will be set to nothing (like ``set var``). When ``--regex`` is used with ``--all``, this behavior changes. Each named variable will contain a list of matches, with the first match contained in the first element, the second match in the second, and so on. If the group was empty or did not match, the corresponding element will be an empty string. If ``--invert`` or ``-v`` is used the selected lines will be only those which do not match the given glob pattern or regular expression. From 9af5b33a6da511dadb37bcc240daac7da42914a7 Mon Sep 17 00:00:00 2001 From: David Adam Date: Fri, 8 Jan 2021 22:12:13 +0800 Subject: [PATCH 075/105] CHANGELOG: work on 3.2.0 --- CHANGELOG.rst | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index a62a8cd51..591bb5669 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -36,9 +36,8 @@ Notable improvements and fixes - ``fish --no-execute`` will no longer complain about unknown commands or non-matching wildcards, as these could be defined differently at runtime (especially for functions). This makes it usable as a static syntax checker (:issue:`977`). -- ``type`` is now a builtin and therefore much faster (:issue:`7342`). - ``string match --regex`` now integrates named PCRE2 capture groups with fish variables, allowing variables to be set directly from ``string match`` (:issue:`7459`). To support this functionality, ``string`` is now a reserved word and can no longer be wrapped in a function. -- Globs and other expansions are limited to 512k results (:issue:`7226`). Because operating systems limit arguments to ARG_MAX, larger values are unlikely to work anyway, and this helps to avoid hangs. +- Globs and other expansions are limited to 512,288 results (:issue:`7226`). Because operating systems limit arguments to ARG_MAX, larger values are unlikely to work anyway, and this helps to avoid hangs. - fish will now always attempt to become process group leader in interactive mode (:issue:`7060`). This helps avoid hangs in certain circumstances, and allows tmux's current directory introspection to work (:issue:`5699`). Syntax changes and new commands @@ -76,7 +75,7 @@ Scripting improvements - ``status`` gained new ``dirname`` and ``basename`` convenience subcommands to get just the directory to the running script or the name of it, to simplify common tasks such as running ``(dirname (status filename))`` (:issue:`7076`). -- The ``_`` gettext function is now implemented as a builtin for performance purposes (:issue:`7036`). +- The ``type`` and ``_`` gettext functions are now implemented as a builtin for performance purposes (:issue:`7342`, :issue:`7036`). - Broken pipelines are now handled more smoothly; in particular, bad redirection mid-pipeline results in the job continuing to run but with the broken file descriptor replaced with a closed file descriptor. This allows better error recovery and is more in line with other shells' @@ -102,7 +101,8 @@ Scripting improvements - ``string`` subcommands now quit early when used with ``--quiet`` (:issue:`7495`). - Failed redirections will now set ``$status`` (:issue:`7540`). - ``read`` can now read interactively from other files, so e.g. forcing it to read from the terminal via ``read `` like ``fish --profile``'s, making the depth more visible (:issue:`7538`). - Resizing the terminal window no longer produces a corrupted prompt (:issue:`6532`). -- ``functions`` produces an error rather than crashing on certain invalid arguments (:issue:`7515`). -- ``fish_private_mode`` may now be changed dynamically using ``set`` (:issue:`7589`). -- Commands with leading spaces may be retrieved from history with up-arrow until a new command is run, matching zsh's ``HIST_IGNORE_SPACE`` (:issue:`1383`). +- ``functions`` produces an error rather than crashing on certain invalid arguments (:issue:`7515`). +- A crash in using tab completions with inline variable assignment (eg ``A= b``) has been fixed (:issue:`7344`). +- ``fish_private_mode`` may now be changed dynamically using ``set`` (:issue:`7589`). +- Commands with leading spaces may be retrieved from history with up-arrow until a new command is run, matching zsh's ``HIST_IGNORE_SPACE`` (:issue:`1383`). +- Importing bash history or reporting errors with recursive globs (``**``) no longer hangs (:issue:`7407`, :issue:`7497`). New or improved bindings ^^^^^^^^^^^^^^^^^^^^^^^^ -- As mentioned above, new readline commands ``undo`` (Control+\_ or Control+Z) and ``redo`` (Alt-/) can be used to revert changes to the command line or the pager search field (:issue:`6570`). +- As mentioned above, new special input functions ``undo`` (Control+\_ or Control+Z) and ``redo`` (Alt-/) can be used to revert changes to the command line or the pager search field (:issue:`6570`). - Control-Z is now available for binding (:issue:`7152`). -- Additionally, using the ``cancel`` readline command (bound to escape by default) right after fish picked an unambiguous completion will undo that (:issue:`7433`). +- Additionally, using the ``cancel`` special input function (bound to escape by default) right after fish picked an unambiguous completion will undo that (:issue:`7433`). - Vi mode bindings now support ``dh``, ``dl``, ``c0``, ``cf``, ``ct``, ``cF``, ``cT``, ``ch``, ``cl``, ``y0``, ``ci``, ``ca``, ``yi``, ``ya``, ``di``, ``da``, ``d;``, ``d,``, ``o``, ``O`` and Control+left/right keys to navigate by word (:issue:`6648`, :issue:`6755`, :issue:`6769`, :issue:`7442`, :issue:`7516`). - Vi mode bindings support ``~`` (tilde) to toggle the case of the selected character (:issue:`6908`). - Functions ``up-or-search`` and ``down-or-search`` (up-arrow and down-arrow) can cross empty lines and don't activate search mode if the search fails which makes it easier to use them to move between lines in some situations. - If history search fails to find a match, the cursor is no longer moved. This is useful when accidentally starting a history search on a multi-line commandline. -- The readline command ``beginning-of-history`` (Page Up) now moves to the oldest search instead of the youngest - that's ``end-of-history`` (Page Down). -- A new readline command ``forward-single-char`` moves one character to the right, and if an autosuggestion is available, only take a single character from it (:issue:`7217`). -- Readline commands can now be joined with ``or`` as a modifier (adding to ``and``), though only some commands report success or failure (:issue:`7217`). +- The special input function ``beginning-of-history`` (Page Up) now moves to the oldest search instead of the youngest - that's ``end-of-history`` (Page Down). +- A new special input function ``forward-single-char`` moves one character to the right, and if an autosuggestion is available, only take a single character from it (:issue:`7217`). +- Special input functions can now be joined with ``or`` as a modifier (adding to ``and``), though only some commands set an exit status (:issue:`7217`). This includes ``suppress-autosuggestion`` to reflect whether an autosuggestion was suppressed (:issue:`1419`) - A new function ``__fish_preview_current_file``, bound to Alt+O, opens the current file at the cursor in a pager (:issue:`6838`). - ``edit_command_buffer`` (Alt-E and Alt-V) passes the cursor position @@ -285,6 +287,9 @@ Deprecations and removed features - fish no longer attempts to modify the terminal size via ``TIOCSWINSZ`` (:issue:`6994`). - The ``fish_color_match`` variable is no longer used. (Previously this controlled the color of matching quotes and parens when using ``read``). - fish 3.2.0 will be the last release in which the redirection to standard error with the ``^`` character is enabled. The ``stderr-nocaret`` feature flag will be changed to "on" in future releases. +- ``string`` is now a reserved word and cannot be used for function names (see above). +- ``fish_vi_cursor``'s option ``--force-iterm`` has been deprecated (see above). +- ``command``, ``jobs`` and ``type`` long-form option ``--quiet`` is deprecated in favor of ``--query`` (see above). For distributors and developers ------------------------------- From 6d1eab9364821c7fb24a9d04eebe1d5ee4feccf8 Mon Sep 17 00:00:00 2001 From: David Adam Date: Fri, 8 Jan 2021 22:22:42 +0800 Subject: [PATCH 076/105] CHANGELOG: fix some Markdown to reStructuredText nits --- CHANGELOG.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 591bb5669..5dacfeb6f 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -2341,7 +2341,7 @@ Other Notable Fixes - Tab completions now work properly within nested subcommands. :issue:`913` -- ``printf`` supports `\e`, the escape character. :issue:`910` +- ``printf`` supports ``\e``, the escape character. :issue:`910` - ``fish_config history`` no longer shows duplicate items. :issue:`900` From bc2612da18696a962c21402312e9f2c75155c4d1 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 8 Jan 2021 18:34:49 +0100 Subject: [PATCH 077/105] CHANGELOG: Some more work of un-issued commits We should really start just adding these to the changelog sooner. --- CHANGELOG.rst | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 5dacfeb6f..ed219b6c9 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -32,6 +32,7 @@ Notable improvements and fixes 1 = 2 and echo true or echo false ^ + This includes numbering the index from 1 instead of 0. - The documentation (:issue:`6500`, :issue:`7371`) and Web-based configuration (:issue:`7523`) received a new theme, matching the design on fishshell.com. - ``fish --no-execute`` will no longer complain about unknown commands or non-matching wildcards, as these could be defined differently at @@ -61,6 +62,7 @@ Scripting improvements - The ``true`` and ``false`` builtins ignore any arguments, like other shells (:issue:`7030`). - Computed ("electric") variables such as ``status`` are now only global in scope, so ``set -Uq status`` returns false (:issue:`7032`). - The output for ``set --show`` has been shortened, only mentioning the scopes in which a variable exists (:issue:`6944`). + In addition it now shows if a variable is a path variable. - A new ``fish_posterror`` event is emitted when attempting to execute a command with syntax errors (:issue:`6880`). - ``fish_indent`` now removes unnecessary quotes in simple cases (:issue:`6722`) and learned a ``--check`` option to just check if a file is indented correctly (:issue:`7251`). @@ -83,6 +85,8 @@ Scripting improvements - ``jobs --quiet PID`` no longer prints "no suitable job" if the job for PID does not exist (eg because it has finished) (:issue:`6809`). - ``command``, ``jobs`` and ``type`` builtins support ``--query`` as the long form of ``-q``, matching other builtins. The long form ``--quiet`` is deprecated (:issue:`7276`). - ``argparse`` no longer requires a short flag letter for long-only options (:issue:`7585`) and only prints a backtrace with invalid options to argparse itself (:issue:`6703`). +- ``argparse`` now passes the validation variables (e.g. ``$_flag_value``) as local-exported variables, + avoiding the need for ``--no-scope-shadowing`` in validation functions. - ``complete`` takes the first argument as the name of the command if the ``--command``/``-c`` option is not used (``complete git`` is treated like ``complete --command git``), and can show the loaded completions for specific commands with ``complete COMMANDNAME`` (:issue:`7321`). - ``set_color -b`` (without an argument) no longer prints an error message, matching other invalid invocations of this command (:issue:`7154`). - Functions triggered by the ``fish_exit`` event are correctly run when the terminal is closed or the shell receives SIGHUP (:issue:`7014`). @@ -100,9 +104,13 @@ Scripting improvements - ``math`` learned tau for those wishing to cut down on typing "2 * pi". - ``string`` subcommands now quit early when used with ``--quiet`` (:issue:`7495`). - Failed redirections will now set ``$status`` (:issue:`7540`). +- More consistent $status after errors, including invalid expansions like ``$foo[``. - ``read`` can now read interactively from other files, so e.g. forcing it to read from the terminal via ``read ``). +- The ``set`` completions no longer hide variables starting with ``__``, they are sorted last instead. Deprecations and removed features --------------------------------- From e93996dc01da56a63aae0dd36544da0f9bf49d99 Mon Sep 17 00:00:00 2001 From: Ben Woods Date: Wed, 6 Jan 2021 00:16:40 +0800 Subject: [PATCH 078/105] completions/pkg: Add support for "alias" and "bootstrap" sub-commands --- share/completions/pkg.fish | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/share/completions/pkg.fish b/share/completions/pkg.fish index 7bdca87f6..5d3cfca24 100644 --- a/share/completions/pkg.fish +++ b/share/completions/pkg.fish @@ -57,10 +57,12 @@ complete -c pkg -n __fish_pkg_subcommand -s 4 -d "Use IPv4" complete -c pkg -n __fish_pkg_subcommand -s 6 -d "Use IPv6" complete -c pkg -n __fish_pkg_subcommand -xa add -d "Install package file" +complete -c pkg -n __fish_pkg_subcommand -xa alias -d "List the command line aliases" complete -c pkg -n __fish_pkg_subcommand -xa annotate -d "Modify annotations on packages" complete -c pkg -n __fish_pkg_subcommand -xa audit -d "Audit installed packages" complete -c pkg -n __fish_pkg_subcommand -xa autoremove -d "Delete unneeded packages" complete -c pkg -n __fish_pkg_subcommand -xa backup -d "Dump package database" +complete -c pkg -n __fish_pkg_subcommand -xa bootstrap -d "Install pkg(8) from remote repository" complete -c pkg -n __fish_pkg_subcommand -xa check -d "Check installed packages" complete -c pkg -n __fish_pkg_subcommand -xa clean -d "Clean local cache" complete -c pkg -n __fish_pkg_subcommand -xa convert -d "Convert package from pkg_add format" @@ -93,15 +95,22 @@ complete -c pkg -n __fish_pkg_subcommand -xa which -d "Check which package provi # add complete -c pkg -n '__fish_pkg_is add install' -s A -l automatic -d "Mark packages as automatic" -complete -c pkg -n '__fish_pkg_is add install' -s f -l force -d "Force installation even when installed" +complete -c pkg -n '__fish_pkg_is add bootstrap install' -s f -l force -d "Force installation even when installed" complete -c pkg -n '__fish_pkg_is add' -s I -l no-scripts -d "Disable installation scripts" complete -c pkg -n '__fish_pkg_is add' -s M -l accept-missing -d "Force installation with missing dependencies" -complete -c pkg -n '__fish_pkg_is add autoremove clean delete remove install update' -s q -l quiet -d "Force quiet output" +complete -c pkg -n '__fish_pkg_is add alias autoremove clean delete remove install update' -s q -l quiet -d "Force quiet output" + +# alias +complete -c pkg -n '__fish_pkg_is alias' -xa '(pkg alias -lq)' +complete -c pkg -n '__fish_pkg_is alias' -s l -l list -d "Print all aliases without their pkg(8) arguments" # autoremove complete -c pkg -n '__fish_pkg_is autoremove clean delete remove install upgrade' -s n -l dry-run -d "Do not make changes" complete -c pkg -n '__fish_pkg_is autoremove clean delete remove install' -s y -l yes -d "Assume yes when asked for confirmation" +# bootstrap +complete -c pkg -n '__fish_pkg_is bootstrap' -f + # check set -l has_check_opt '__fish_contains_opt -s B shlibs -s d dependencies -s s checksums -s r recompute' set -l has_all_opt '__fish_contains_opt -s a all' From bee8e8f6f716e37c216ce82b6252016fd753b83f Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Thu, 31 Dec 2020 13:30:58 -0800 Subject: [PATCH 079/105] Expand more when performing history path detection When adding a command to history, we first expand its arguments to see if any arguments are paths which refer to files. If so, we will only autosuggest that command from history if the files are still valid. For example, if the user runs `rm ./file.txt` then we will remember that `./file.txt` referred to a file, and then only autosuggest that if the file is present again. Prior to this change we only performed simple expansion relative to the working directory. This change extends it to variables and tilde expansion. For example we will now apply the same hinting for `rm ~/file.txt` Fixes #7582 --- src/fish_tests.cpp | 24 ++++++++++++-------- src/highlight.cpp | 2 +- src/history.cpp | 49 +++++++++++++++++++++++++++++------------ src/history.h | 24 ++++++++++++-------- src/operation_context.h | 5 +++-- src/reader.cpp | 2 +- 6 files changed, 70 insertions(+), 36 deletions(-) diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 2784e2b72..05c4017df 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -4353,17 +4353,23 @@ void history_tests_t::test_history_path_detection() { } fclose(f); - test_environment_t vars; - vars.vars[L"PWD"] = tmpdir; - vars.vars[L"HOME"] = tmpdir; + std::shared_ptr vars = std::make_shared(); + vars->vars[L"PWD"] = tmpdir; + vars->vars[L"HOME"] = tmpdir; history_t &history = history_t::history_with_name(L"path_detection"); - history.add_pending_with_file_detection(L"cmd0 not/a/valid/path", tmpdir); - history.add_pending_with_file_detection(L"cmd1 " + filename, tmpdir); - history.add_pending_with_file_detection(L"cmd2 " + tmpdir + L"/" + filename, tmpdir); + history.add_pending_with_file_detection(L"cmd0 not/a/valid/path", vars); + history.add_pending_with_file_detection(L"cmd1 " + filename, vars); + history.add_pending_with_file_detection(L"cmd2 " + tmpdir + L"/" + filename, vars); + history.add_pending_with_file_detection(L"cmd3 $HOME/" + filename, vars); + history.add_pending_with_file_detection(L"cmd4 $HOME/notafile", vars); + history.add_pending_with_file_detection(L"cmd5 ~/" + filename, vars); + history.add_pending_with_file_detection(L"cmd6 ~/notafile", vars); + history.add_pending_with_file_detection(L"cmd7 ~/*f*", vars); + history.add_pending_with_file_detection(L"cmd8 ~/*zzz*", vars); history.resolve_pending(); - constexpr size_t hist_size = 3; + constexpr size_t hist_size = 9; if (history.size() != hist_size) { err(L"history has wrong size: %lu but expected %lu", (unsigned long)history.size(), (unsigned long)hist_size); history.clear(); @@ -4372,9 +4378,9 @@ void history_tests_t::test_history_path_detection() { // Expected sets of paths. wcstring_list_t expected[hist_size] = { + {}, {filename}, {tmpdir + L"/" + filename}, {L"$HOME/" + filename}, {}, {L"~/" + filename}, + {}, {}, // we do not expand globs {}, - {filename}, - {tmpdir + L"/" + filename}, }; size_t lap; diff --git a/src/highlight.cpp b/src/highlight.cpp index 444b9a4d1..b092b04aa 100644 --- a/src/highlight.cpp +++ b/src/highlight.cpp @@ -468,7 +468,7 @@ bool autosuggest_validate_from_history(const history_item_t &item, } // Did the historical command have arguments that look like paths, which aren't paths now? - if (!all_paths_are_valid(item.get_required_paths(), working_directory)) { + if (!all_paths_are_valid(item.get_required_paths(), ctx)) { return false; } diff --git a/src/history.cpp b/src/history.cpp index 1ae664cb5..647176094 100644 --- a/src/history.cpp +++ b/src/history.cpp @@ -369,7 +369,7 @@ struct history_impl_t { std::unordered_map items_at_indexes(const std::vector &idxs); // Sets the valid file paths for the history item with the given identifier. - void set_valid_file_paths(const wcstring_list_t &valid_file_paths, history_identifier_t ident); + void set_valid_file_paths(wcstring_list_t &&valid_file_paths, history_identifier_t ident); // Return the specified history at the specified index. 0 is the index of the current // commandline. (So the most recent item is at index 1.) @@ -459,7 +459,7 @@ void history_impl_t::remove(const wcstring &str_to_remove) { assert(first_unwritten_new_item_index <= new_items.size()); } -void history_impl_t::set_valid_file_paths(const wcstring_list_t &valid_file_paths, +void history_impl_t::set_valid_file_paths(wcstring_list_t &&valid_file_paths, history_identifier_t ident) { // 0 identifier is used to mean "not necessary". if (ident == 0) { @@ -469,7 +469,7 @@ void history_impl_t::set_valid_file_paths(const wcstring_list_t &valid_file_path // Look for an item with the given identifier. It is likely to be at the end of new_items. for (auto iter = new_items.rbegin(); iter != new_items.rend(); ++iter) { if (iter->identifier == ident) { // found it - iter->required_paths = valid_file_paths; + iter->required_paths = std::move(valid_file_paths); break; } } @@ -1246,21 +1246,42 @@ wcstring history_session_id(const environment_t &vars) { return result; } -path_list_t valid_paths(const path_list_t &paths, const wcstring &working_directory) { +path_list_t expand_and_detect_paths(const path_list_t &paths, const environment_t &vars) { ASSERT_IS_BACKGROUND_THREAD(); wcstring_list_t result; + wcstring working_directory = vars.get_pwd_slash(); + operation_context_t ctx(vars, kExpansionLimitBackground); for (const wcstring &path : paths) { - if (path_is_valid(path, working_directory)) { - result.push_back(path); + // Suppress cmdsubs since we are on a background thread and don't want to execute fish + // script. + // Suppress wildcards because we want to suggest e.g. `rm *` even if the directory + // is empty (and so rm will fail); this is nevertheless a useful command because it + // confirms the directory is empty. + wcstring expanded_path = path; + if (expand_one(expanded_path, {expand_flag::skip_cmdsubst, expand_flag::skip_wildcards}, + ctx)) { + if (path_is_valid(expanded_path, working_directory)) { + // Note we return the original (unexpanded) path. + result.push_back(path); + } } } return result; } -bool all_paths_are_valid(const path_list_t &paths, const wcstring &working_directory) { +bool all_paths_are_valid(const path_list_t &paths, const operation_context_t &ctx) { ASSERT_IS_BACKGROUND_THREAD(); + wcstring working_directory = ctx.vars.get_pwd_slash(); for (const wcstring &path : paths) { - if (!path_is_valid(path, working_directory)) { + if (ctx.cancel_checker()) { + return false; + } + wcstring expanded_path = path; + if (!expand_one(expanded_path, {expand_flag::skip_cmdsubst, expand_flag::skip_wildcards}, + ctx)) { + return false; + } + if (!path_is_valid(expanded_path, working_directory)) { return false; } } @@ -1304,7 +1325,7 @@ void history_t::remove(const wcstring &str) { impl()->remove(str); } void history_t::remove_ephemeral_items() { impl()->remove_ephemeral_items(); } void history_t::add_pending_with_file_detection(const wcstring &str, - const wcstring &working_dir_slash, + const std::shared_ptr &vars, history_persistence_mode_t persist_mode) { // We use empty items as sentinels to indicate the end of history. // Do not allow them to be added (#6032). @@ -1321,9 +1342,8 @@ void history_t::add_pending_with_file_detection(const wcstring &str, for (const node_t &node : ast) { if (const argument_t *arg = node.try_as()) { wcstring potential_path = arg->source(str); - bool unescaped = unescape_string_in_place(&potential_path, UNESCAPE_DEFAULT); - if (unescaped && string_could_be_path(potential_path)) { - potential_paths.push_back(potential_path); + if (string_could_be_path(potential_path)) { + potential_paths.push_back(std::move(potential_path)); } } else if (const decorated_statement_t *stmt = node.try_as()) { // Hack hack hack - if the command is likely to trigger an exit, then don't do @@ -1362,9 +1382,10 @@ void history_t::add_pending_with_file_detection(const wcstring &str, // Don't hold the lock while we perform this file detection. imp->add(std::move(item), true /* pending */); iothread_perform([=]() { - auto validated_paths = valid_paths(potential_paths, working_dir_slash); + // Don't hold the lock while we perform this file detection. + auto validated_paths = expand_and_detect_paths(potential_paths, *vars); auto imp = this->impl(); - imp->set_valid_file_paths(validated_paths, identifier); + imp->set_valid_file_paths(std::move(validated_paths), identifier); imp->enable_automatic_saving(); }); } else { diff --git a/src/history.h b/src/history.h index 7506f62ab..9f5e24c6c 100644 --- a/src/history.h +++ b/src/history.h @@ -24,6 +24,7 @@ struct io_streams_t; class env_stack_t; class environment_t; +class operation_context_t; // Fish supports multiple shells writing to history at once. Here is its strategy: // @@ -178,9 +179,10 @@ class history_t { void remove_ephemeral_items(); // Add a new pending history item to the end, and then begin file detection on the items to - // determine which arguments are paths. The item has the given \p persist_mode. + // determine which arguments are paths. Arguments may be expanded (e.g. with PWD and variables) + // using the given \p vars. The item has the given \p persist_mode void add_pending_with_file_detection( - const wcstring &str, const wcstring &working_dir_slash, + const wcstring &str, const std::shared_ptr &vars, history_persistence_mode_t persist_mode = history_persistence_mode_t::disk); // Resolves any pending history items, so that they may be returned in history searches. @@ -301,14 +303,18 @@ void history_save_all(); /// Return the prefix for the files to be used for command and read history. wcstring history_session_id(const environment_t &vars); -/// Given a list of paths and a working directory, return the paths that are valid -/// This does disk I/O and may only be called in a background thread -path_list_t valid_paths(const path_list_t &paths, const wcstring &working_directory); +/// Given a list of proposed paths and a context, perform variable and home directory expansion, +/// and detect if the result expands to a value which is also the path to a file. +/// Wildcard expansions are suppressed - see implementation comments for why. +/// This is used for autosuggestion hinting. If we add an item to history, and one of its arguments +/// refers to a file, then we only want to suggest it if there is a valid file there. +/// This does disk I/O and may only be called in a background thread. +path_list_t expand_and_detect_paths(const path_list_t &paths, const environment_t &vars); -/// Given a list of paths and a working directory, -/// return true if all paths in the list are valid -/// Returns true for if paths is empty -bool all_paths_are_valid(const path_list_t &paths, const wcstring &working_directory); +/// Given a list of proposed paths and a context, expand each one and see if it refers to a file. +/// Wildcard expansions are suppressed. +/// \return true if \p paths is empty or every path is valid. +bool all_paths_are_valid(const path_list_t &paths, const operation_context_t &ctx); /// Sets private mode on. Once in private mode, it cannot be turned off. void start_private_mode(env_stack_t &vars); diff --git a/src/operation_context.h b/src/operation_context.h index 56b731d99..6f89853e6 100644 --- a/src/operation_context.h +++ b/src/operation_context.h @@ -60,8 +60,9 @@ class operation_context_t { size_t expansion_limit = kExpansionLimitDefault); /// Construct from vars alone. - explicit operation_context_t(const environment_t &vars) - : operation_context_t(nullptr, vars, no_cancel) {} + explicit operation_context_t(const environment_t &vars, + size_t expansion_limit = kExpansionLimitDefault) + : operation_context_t(nullptr, vars, no_cancel, expansion_limit) {} ~operation_context_t(); }; diff --git a/src/reader.cpp b/src/reader.cpp index 2c4b8e5b3..3b3f1a514 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -3241,7 +3241,7 @@ void reader_data_t::handle_readline_command(readline_cmd_t c, readline_loop_stat } else { mode = history_persistence_mode_t::disk; } - history->add_pending_with_file_detection(text, vars.get_pwd_slash(), mode); + history->add_pending_with_file_detection(text, vars.snapshot(), mode); } rls.finished = true; From 6f91195f403c9e50f2fa44baab6e3bfad19c44ce Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Fri, 8 Jan 2021 14:14:05 -0800 Subject: [PATCH 080/105] Stop using unique_ptr to store histories These register shutdown dtors, which cause tsan to complain. --- src/history.cpp | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/history.cpp b/src/history.cpp index 647176094..2bf4fed8e 100644 --- a/src/history.cpp +++ b/src/history.cpp @@ -1481,8 +1481,8 @@ history_item_t history_t::item_at_index(size_t idx) { return impl()->item_at_ind size_t history_t::size() { return impl()->size(); } -/// The set of all histories. -static owning_lock>> s_histories; +/// The set of all histories. These are deliberately leaked to avoid shutdown dtors. +static owning_lock> s_histories; void history_save_all() { auto histories = s_histories.acquire(); @@ -1492,13 +1492,13 @@ void history_save_all() { } history_t &history_t::history_with_name(const wcstring &name) { - // Return a history for the given name, creating it if necessary + // Return a history for the given name, creating it if necessary. // Note that histories are currently never deleted, so we can return a reference to them without - // using something like shared_ptr + // using something like shared_ptr. auto hs = s_histories.acquire(); - std::unique_ptr &hist = (*hs)[name]; + history_t *&hist = (*hs)[name]; if (!hist) { - hist = make_unique(name); + hist = new history_t(name); } return *hist; } From 3c3d09b65fe9736b4282d53bbd098897fba3b9f9 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Fri, 8 Jan 2021 19:36:56 -0800 Subject: [PATCH 081/105] Fix a tsan warning in features_t --- src/future_feature_flags.h | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/src/future_feature_flags.h b/src/future_feature_flags.h index 0140ea82a..e31b306e7 100644 --- a/src/future_feature_flags.h +++ b/src/future_feature_flags.h @@ -4,6 +4,7 @@ #include +#include #include #include "common.h" @@ -28,13 +29,13 @@ class features_t { /// Return whether a flag is set. bool test(flag_t f) const { assert(f >= 0 && f < flag_count && "Invalid flag"); - return values[f]; + return values[f].load(std::memory_order_relaxed); } /// Set a flag. void set(flag_t f, bool value) { assert(f >= 0 && f < flag_count && "Invalid flag"); - values[f] = value; + values[f].store(value, std::memory_order_relaxed); } /// Parses a comma-separated feature-flag string, updating ourselves with the values. @@ -69,9 +70,20 @@ class features_t { features_t(); + features_t(const features_t &rhs) { *this = rhs; } + + void operator=(const features_t &rhs) { + for (int i = 0; i < flag_count; i++) { + flag_t f = static_cast(i); + this->set(f, rhs.test(f)); + } + } + private: - /// Values for the flags. - bool values[flag_count] = {}; + // Values for the flags. + // These are atomic to "fix" a race reported by tsan where tests of feature flags and other + // tests which use them conceptually race. + std::atomic values[flag_count]{}; }; /// Return the global set of features for fish. This is const to prevent accidental mutation. From 1dd776ec990a5317326dea9b408f0975db596c23 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sat, 9 Jan 2021 08:43:07 +0100 Subject: [PATCH 082/105] echo: Don't interpret and print options A weird interaction between grouped short options and our weird option parsing that puts unknown options back: ``` echo "-n foo" ``` would see the `-n`, turn off printing newlines, interpret the " " as another grouped short option, see that there is no short option for space and put the entire token back on the arguments pile. So it would print "-n foo" *without a newline*. Fix this by keeping an old state of the options around and reverting it when putting options back. The alternative is *probably* to forbid the " " short option in wgetopt, then check if an option group contains it and error out, but this should only really be a problem in `echo` because that is, AFAICT, the only thing that puts the options back. Fixes #7614 --- src/builtin_echo.cpp | 14 ++++++++++++++ tests/checks/basic.fish | 9 +++++++++ 2 files changed, 23 insertions(+) diff --git a/src/builtin_echo.cpp b/src/builtin_echo.cpp index 9d2c78e71..55051f689 100644 --- a/src/builtin_echo.cpp +++ b/src/builtin_echo.cpp @@ -28,6 +28,8 @@ static int parse_cmd_opts(echo_cmd_opts_t &opts, int *optind, int argc, wchar_t wchar_t *cmd = argv[0]; int opt; wgetopter_t w; + echo_cmd_opts_t oldopts = opts; + int oldoptind = 0; while ((opt = w.wgetopt_long(argc, argv, short_options, long_options, nullptr)) != -1) { switch (opt) { case 'n': { @@ -51,6 +53,7 @@ static int parse_cmd_opts(echo_cmd_opts_t &opts, int *optind, int argc, wchar_t return STATUS_INVALID_ARGS; } case '?': { + opts = oldopts; *optind = w.woptind - 1; return STATUS_CMD_OK; } @@ -58,6 +61,17 @@ static int parse_cmd_opts(echo_cmd_opts_t &opts, int *optind, int argc, wchar_t DIE("unexpected retval from wgetopt_long"); } } + + // Super cheesy: We keep an old copy of the option state around, + // so we can revert it in case we get an argument like + // "-n foo". + // We need to keep it one out-of-date so we can ignore the *last* option. + // (this might be an issue in wgetopt, but that's a whole other can of worms + // and really only occurs with our weird "put it back" option parsing) + if (w.woptind == oldoptind + 2) { + oldopts = opts; + oldoptind = w.woptind; + } } *optind = w.woptind; diff --git a/tests/checks/basic.fish b/tests/checks/basic.fish index 2c231040b..c16ab5d7e 100644 --- a/tests/checks/basic.fish +++ b/tests/checks/basic.fish @@ -475,3 +475,12 @@ echo $status builtin --query echo echo $status #CHECK: 0 + +# Check that echo doesn't interpret options *and print them* +# at the start of quoted args: +echo '-ne \tart' +# CHECK: -ne \tart +echo '-n art' +echo banana +# CHECK: -n art +# CHECK: banana From f22fe44c79486e0992005e1fc6e3360fc6e9f471 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sat, 9 Jan 2021 12:13:24 +0100 Subject: [PATCH 083/105] CHANGELOG 7614 --- CHANGELOG.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index ed219b6c9..463f5e0e9 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -111,6 +111,7 @@ Scripting improvements - ``.`` and ``:`` are now also builtins instead of functions (:issue:`6854`). - ``functions`` now explains when a function was defined via ``source`` instead of just saying ``Defined in -``. - Significant performance improvements when globbing or in ``math``. +- ``echo`` no longer interprets options at the beginning of an argument (``echo "-n foo"``) (:issue:`7614`). Interactive improvements ------------------------ From b489137fa9d31bc70754168600d63af8beb5bdf0 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sat, 9 Jan 2021 13:13:48 +0100 Subject: [PATCH 084/105] docs: Link to fish_key_reader --- doc_src/cmds/bind.rst | 2 ++ doc_src/index.rst | 1 + 2 files changed, 3 insertions(+) diff --git a/doc_src/cmds/bind.rst b/doc_src/cmds/bind.rst index 4c5be95f7..acfdc07eb 100644 --- a/doc_src/cmds/bind.rst +++ b/doc_src/cmds/bind.rst @@ -28,6 +28,8 @@ The generic key binding that matches if no other binding does can be set by spec If the ``-k`` switch is used, the name of a key (such as 'down', 'up' or 'backspace') is used instead of a sequence. The names used are the same as the corresponding curses variables, but without the 'key\_' prefix. (See ``terminfo(5)`` for more information, or use ``bind --key-names`` for a list of all available named keys). Normally this will print an error if the current ``$TERM`` entry doesn't have a given key, unless the ``-s`` switch is given. +To find out what sequence a key combination sends, you can use :ref:`fish_key_reader `. + ``COMMAND`` can be any fish command, but it can also be one of a set of special input functions. These include functions for moving the cursor, operating on the kill-ring, performing tab completion, etc. Use ``bind --function-names`` for a complete list of these input functions. When ``COMMAND`` is a shellscript command, it is a good practice to put the actual code into a `function <#function>`__ and simply bind to the function name. This way it becomes significantly easier to test the function while editing, and the result is usually more readable as well. diff --git a/doc_src/index.rst b/doc_src/index.rst index 7c582cf6e..cd399e751 100644 --- a/doc_src/index.rst +++ b/doc_src/index.rst @@ -229,6 +229,7 @@ Some characters can not be written directly on the command line. For these chara - ``\cX``, where *X* is a letter of the alphabet, represents the control sequence generated by pressing the control key and the specified letter. For example, ``\ci`` is the tab character +These sequences are also used with the :ref:`bind ` builtin to describe key sequences. To see what sequence a key combination sends and how to write it, use :ref:`fish_key_reader `. .. _redirects: From 168677f8b354485b656897bf43a387f8a3e3aa75 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sat, 9 Jan 2021 13:44:48 +0100 Subject: [PATCH 085/105] Use disown with $last_pid As mentioned in 5b706faa731ee073968dfa1d7b1f73375438c608, bare `disown` has a problem: It disowns the last *existing* job. Unfortunately, it's easy to see cases where that won't happen: sleep 5m & /bin/true & # will exit immediately disown # will most likely disown *sleep*, not true So what we do is to pass $last_pid. In help especially this is likely to occur because many graphical browsers fork immediately to avoid blocking the terminal (we only added the backgrounding and disown because some weren't). Note that it's *possible* this doesn't occur if used in the same function, but I don't want to rely on those semantics. It might be worth doing this as the default - see #7210. --- share/functions/__fish_config_interactive.fish | 2 +- share/functions/help.fish | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/share/functions/__fish_config_interactive.fish b/share/functions/__fish_config_interactive.fish index 35c557f25..75930e5f0 100644 --- a/share/functions/__fish_config_interactive.fish +++ b/share/functions/__fish_config_interactive.fish @@ -94,7 +94,7 @@ function __fish_config_interactive -d "Initializations that should be performed # Run python directly in the background and swallow all output $python $update_args >/dev/null 2>&1 & # Then disown the job so that it continues to run in case of an early exit (#6269) - disown >/dev/null 2>&1 + disown $last_pid >/dev/null 2>&1 end end end diff --git a/share/functions/help.fish b/share/functions/help.fish index 857a2f499..51debe4f3 100644 --- a/share/functions/help.fish +++ b/share/functions/help.fish @@ -199,7 +199,7 @@ function help --description 'Show help for the fish shell' printf (_ 'help: Help is being displayed in %s.\n') $fish_browser[1] end $fish_browser $page_url & - disown + disown $last_pid >/dev/null 2>&1 else # Work around lynx bug where
always has the same formatting as links (unreadable) # by using a custom style sheet. See https://github.com/fish-shell/fish-shell/issues/4170 From 89687e7db7c2ffec0ed90f32e9cf1ad3268cea89 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Sat, 9 Jan 2021 13:14:54 -0800 Subject: [PATCH 086/105] Fix a warning building on Linux Initialize saved_errno --- src/env_universal_common.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/env_universal_common.cpp b/src/env_universal_common.cpp index 965533287..4278d18b9 100644 --- a/src/env_universal_common.cpp +++ b/src/env_universal_common.cpp @@ -535,7 +535,7 @@ autoclose_fd_t env_universal_t::open_temporary_file(const wcstring &directory, w // This should almost always succeed on the first try. assert(!string_suffixes_string(L"/", directory)); //!OCLINT(multiple unary operator) - int saved_errno; + int saved_errno = 0; const wcstring tmp_name_template = directory + L"/fishd.tmp.XXXXXX"; autoclose_fd_t result; std::string narrow_str; From 7fc72e46b387a1b0fd1af2dc680a8e9217ac0791 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sat, 9 Jan 2021 22:44:30 +0100 Subject: [PATCH 087/105] docs: Add more about $PATH being imported to the tutorial Fixes #7539. --- doc_src/tutorial.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc_src/tutorial.rst b/doc_src/tutorial.rst index 42675cd28..571142aad 100644 --- a/doc_src/tutorial.rst +++ b/doc_src/tutorial.rst @@ -275,6 +275,8 @@ You can erase (or "delete") a variable with ``-e`` or ``--erase`` > env | grep MyVariable (no output) +.. _tut-exports: + Exports (Shell Variables) ------------------------- @@ -290,6 +292,7 @@ To give a variable to an external command, it needs to be "exported". Unlike oth It can also be unexported with ``--unexport`` or ``-u``. +This works the other way around as well! If fish is started by something else, it inherits that parents exported variables. So if your terminal emulator starts fish, and it exports ``$LANG`` set to ``en_US.UTF-8``, fish will receive that setting. And whatever started your terminal emulator also gave *it* some variables that it will then pass on unless it specifically decides not to. This is how fish usually receives the values for things like ``$LANG``, ``$PATH`` and ``$TERM``, without you having to specify them again. .. _tut-lists: @@ -641,6 +644,8 @@ $PATH ``$PATH`` is an environment variable containing the directories that fish searches for commands. Unlike other shells, $PATH is a :ref:`list `, not a colon-delimited string. +Fish takes care to set ``$PATH`` to a default, but typically it is just inherited from fish's parent process and is set to a value that makes sense for the system - see :ref:`Exports `. + To prepend /usr/local/bin and /usr/sbin to ``$PATH``, you can write:: > set PATH /usr/local/bin /usr/sbin $PATH From 884eb2b1986262c65a52159b214a359a155ceb30 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Sat, 9 Jan 2021 14:00:01 -0800 Subject: [PATCH 088/105] Remove an unused static variable --- src/history.cpp | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/history.cpp b/src/history.cpp index 2bf4fed8e..9ae8d8886 100644 --- a/src/history.cpp +++ b/src/history.cpp @@ -1503,8 +1503,6 @@ history_t &history_t::history_with_name(const wcstring &name) { return *hist; } -static relaxed_atomic_bool_t private_mode{false}; - void start_private_mode(env_stack_t &vars) { vars.set_one(L"fish_history", ENV_GLOBAL, L""); vars.set_one(L"fish_private_mode", ENV_GLOBAL, L"1"); From 87dacc0e9536f49bfbd50983bb18e8f0c6235eac Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Sat, 9 Jan 2021 15:02:21 -0800 Subject: [PATCH 089/105] Improve formatting and layout of history path detection test --- src/fish_tests.cpp | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 05c4017df..57fcef88d 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -4378,9 +4378,15 @@ void history_tests_t::test_history_path_detection() { // Expected sets of paths. wcstring_list_t expected[hist_size] = { - {}, {filename}, {tmpdir + L"/" + filename}, {L"$HOME/" + filename}, {}, {L"~/" + filename}, - {}, {}, // we do not expand globs - {}, + {}, // cmd0 + {filename}, // cmd1 + {tmpdir + L"/" + filename}, // cmd2 + {L"$HOME/" + filename}, // cmd3 + {}, // cmd4 + {L"~/" + filename}, // cmd5 + {}, // cmd6 + {}, // cmd7 - we do not expand globs + {}, // cmd8 }; size_t lap; From e062a07a97368db29bfc5eea1675bdcedf5c6d20 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Sat, 9 Jan 2021 14:40:17 -0800 Subject: [PATCH 090/105] Revert "Stop using unique_ptr to store histories" This reverts commit 6f91195f403c9e50f2fa44baab6e3bfad19c44ce. This triggered ASan complaints due to leaks. --- src/history.cpp | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/history.cpp b/src/history.cpp index 9ae8d8886..830d1a2d4 100644 --- a/src/history.cpp +++ b/src/history.cpp @@ -1481,8 +1481,8 @@ history_item_t history_t::item_at_index(size_t idx) { return impl()->item_at_ind size_t history_t::size() { return impl()->size(); } -/// The set of all histories. These are deliberately leaked to avoid shutdown dtors. -static owning_lock> s_histories; +/// The set of all histories. +static owning_lock>> s_histories; void history_save_all() { auto histories = s_histories.acquire(); @@ -1492,13 +1492,13 @@ void history_save_all() { } history_t &history_t::history_with_name(const wcstring &name) { - // Return a history for the given name, creating it if necessary. + // Return a history for the given name, creating it if necessary // Note that histories are currently never deleted, so we can return a reference to them without - // using something like shared_ptr. + // using something like shared_ptr auto hs = s_histories.acquire(); - history_t *&hist = (*hs)[name]; + std::unique_ptr &hist = (*hs)[name]; if (!hist) { - hist = new history_t(name); + hist = make_unique(name); } return *hist; } From e8c9da100c438b1876fb04fade314e67f92f690d Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Sat, 9 Jan 2021 16:22:32 -0800 Subject: [PATCH 091/105] Track histories with shared_ptr Prior to this change, histories were immortal and allocated with either unique_ptr or just leaked via new. But this can result in races in the path detection test, as the destructor races with the pointer-captured history. Switch to using shared_ptr. --- src/builtin_history.cpp | 4 +- src/builtin_set.cpp | 4 +- src/complete.cpp | 4 +- src/env.cpp | 4 +- src/expand.cpp | 4 +- src/fish_tests.cpp | 86 +++++++++++++++++++++-------------------- src/history.cpp | 29 +++++++------- src/history.h | 23 +++++++---- src/reader.cpp | 25 ++++++------ src/reader.h | 2 +- 10 files changed, 100 insertions(+), 85 deletions(-) diff --git a/src/builtin_history.cpp b/src/builtin_history.cpp index 61716a77f..5dffd9e98 100644 --- a/src/builtin_history.cpp +++ b/src/builtin_history.cpp @@ -216,8 +216,8 @@ maybe_t builtin_history(parser_t &parser, io_streams_t &streams, wchar_t ** // Use the default history if we have none (which happens if invoked non-interactively, e.g. // from webconfig.py. - history_t *history = reader_get_history(); - if (!history) history = &history_t::history_with_name(history_session_id(parser.vars())); + std::shared_ptr history = reader_get_history(); + if (!history) history = history_t::with_name(history_session_id(parser.vars())); // If a history command hasn't already been specified via a flag check the first word. // Note that this can be simplified after we eliminate allowing subcommands as flags. diff --git a/src/builtin_set.cpp b/src/builtin_set.cpp index c4d3e8b90..7e5aca16f 100644 --- a/src/builtin_set.cpp +++ b/src/builtin_set.cpp @@ -483,8 +483,8 @@ static int builtin_set_list(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, if (!names_only) { wcstring val; if (opts.shorten_ok && key == L"history") { - history_t *history = - &history_t::history_with_name(history_session_id(parser.vars())); + std::shared_ptr history = + history_t::with_name(history_session_id(parser.vars())); for (size_t i = 1; i < history->size() && val.size() < 64; i++) { if (i > 1) val += L' '; val += expand_escape_string(history->item_at_index(i).str()); diff --git a/src/complete.cpp b/src/complete.cpp index 457cd8b90..c37b24ae4 100644 --- a/src/complete.cpp +++ b/src/complete.cpp @@ -1252,8 +1252,8 @@ bool completer_t::complete_variable(const wcstring &str, size_t start_offset) { // $history can be huge, don't put all of it in the completion description; see // #6288. if (env_name == L"history") { - history_t *history = - &history_t::history_with_name(history_session_id(ctx.vars)); + std::shared_ptr history = + history_t::with_name(history_session_id(ctx.vars)); for (size_t i = 1; i < history->size() && desc.size() < 64; i++) { if (i > 1) desc += L' '; desc += expand_escape_string(history->item_at_index(i).str()); diff --git a/src/env.cpp b/src/env.cpp index 80fadb131..99fd3c7ca 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -687,9 +687,9 @@ maybe_t env_scoped_impl_t::try_get_computed(const wcstring &key) cons return none(); } - history_t *history = reader_get_history(); + std::shared_ptr history = reader_get_history(); if (!history) { - history = &history_t::history_with_name(history_session_id(*this)); + history = history_t::with_name(history_session_id(*this)); } wcstring_list_t result; if (history) history->get_history(result); diff --git a/src/expand.cpp b/src/expand.cpp index fd4b696a9..091ffbc5d 100644 --- a/src/expand.cpp +++ b/src/expand.cpp @@ -355,13 +355,13 @@ static expand_result_t expand_variables(wcstring instr, completion_receiver_t *o // Do a dirty hack to make sliced history fast (#4650). We expand from either a variable, or a // history_t. Note that "history" is read only in env.cpp so it's safe to special-case it in // this way (it cannot be shadowed, etc). - history_t *history = nullptr; + std::shared_ptr history{}; maybe_t var{}; if (var_name == L"history") { // Note reader_get_history may return null, if we are running non-interactively (e.g. from // web_config). if (is_main_thread()) { - history = &history_t::history_with_name(history_session_id(env_stack_t::principal())); + history = history_t::with_name(history_session_id(env_stack_t::principal())); } } else if (var_name != wcstring{VARIABLE_EXPAND_EMPTY}) { var = vars.get(var_name); diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 57fcef88d..0331cf77c 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -3508,7 +3508,7 @@ static bool history_contains(history_t *history, const wcstring &txt) { return result; } -static bool history_contains(const std::unique_ptr &history, const wcstring &txt) { +static bool history_contains(const std::shared_ptr &history, const wcstring &txt) { return history_contains(history.get(), txt); } @@ -4007,10 +4007,10 @@ void history_tests_t::test_history() { const history_search_flags_t nocase = history_search_ignore_case; // Populate a history. - history_t &history = history_t::history_with_name(L"test_history"); - history.clear(); + std::shared_ptr history = history_t::with_name(L"test_history"); + history->clear(); for (const wcstring &s : items) { - history.add(s); + history->add(s); } // Helper to set expected items to those matching a predicate, in reverse order. @@ -4062,13 +4062,13 @@ void history_tests_t::test_history() { // Test item removal case-sensitive. searcher = history_search_t(history, L"Alpha"); test_history_matches(searcher, {L"Alpha"}, __LINE__); - history.remove(L"Alpha"); + history->remove(L"Alpha"); searcher = history_search_t(history, L"Alpha"); test_history_matches(searcher, {}, __LINE__); // Test history escaping and unescaping, yaml, etc. history_item_list_t before, after; - history.clear(); + history->clear(); size_t i, max = 100; for (i = 1; i <= max; i++) { // Generate a value. @@ -4088,17 +4088,17 @@ void history_tests_t::test_history() { history_item_t item(value, time(NULL)); item.required_paths = paths; before.push_back(item); - history.add(item); + history->add(item); } - history.save(); + history->save(); // Empty items should just be dropped (#6032). - history.add(L""); - do_test(!history.item_at_index(1).contents.empty()); + history->add(L""); + do_test(!history->item_at_index(1).contents.empty()); // Read items back in reverse order and ensure they're the same. for (i = 100; i >= 1; i--) { - history_item_t item = history.item_at_index(i); + history_item_t item = history->item_at_index(i); do_test(!item.empty()); after.push_back(item); } @@ -4111,7 +4111,7 @@ void history_tests_t::test_history() { } // Clean up after our tests. - history.clear(); + history->clear(); } // Wait until the next second. @@ -4245,8 +4245,9 @@ void history_tests_t::test_history_merge() { say(L"Testing history merge"); const size_t count = 3; const wcstring name = L"merge_test"; - std::unique_ptr hists[count] = { - make_unique(name), make_unique(name), make_unique(name)}; + std::shared_ptr hists[count] = {std::make_shared(name), + std::make_shared(name), + std::make_shared(name)}; const wcstring texts[count] = {L"History 1", L"History 2", L"History 3"}; const wcstring alt_texts[count] = {L"History Alt 1", L"History Alt 2", L"History Alt 3"}; @@ -4280,7 +4281,7 @@ void history_tests_t::test_history_merge() { // Make a new history. It should contain everything. The time_barrier() is so that the timestamp // is newer, since we only pick up items whose timestamp is before the birth stamp. time_barrier(); - std::unique_ptr everything = make_unique(name); + std::shared_ptr everything = std::make_shared(name); for (const auto &text : texts) { do_test(history_contains(everything, text)); } @@ -4357,22 +4358,23 @@ void history_tests_t::test_history_path_detection() { vars->vars[L"PWD"] = tmpdir; vars->vars[L"HOME"] = tmpdir; - history_t &history = history_t::history_with_name(L"path_detection"); - history.add_pending_with_file_detection(L"cmd0 not/a/valid/path", vars); - history.add_pending_with_file_detection(L"cmd1 " + filename, vars); - history.add_pending_with_file_detection(L"cmd2 " + tmpdir + L"/" + filename, vars); - history.add_pending_with_file_detection(L"cmd3 $HOME/" + filename, vars); - history.add_pending_with_file_detection(L"cmd4 $HOME/notafile", vars); - history.add_pending_with_file_detection(L"cmd5 ~/" + filename, vars); - history.add_pending_with_file_detection(L"cmd6 ~/notafile", vars); - history.add_pending_with_file_detection(L"cmd7 ~/*f*", vars); - history.add_pending_with_file_detection(L"cmd8 ~/*zzz*", vars); - history.resolve_pending(); + std::shared_ptr history = history_t::with_name(L"path_detection"); + history_t::add_pending_with_file_detection(history, L"cmd0 not/a/valid/path", vars); + history_t::add_pending_with_file_detection(history, L"cmd1 " + filename, vars); + history_t::add_pending_with_file_detection(history, L"cmd2 " + tmpdir + L"/" + filename, vars); + history_t::add_pending_with_file_detection(history, L"cmd3 $HOME/" + filename, vars); + history_t::add_pending_with_file_detection(history, L"cmd4 $HOME/notafile", vars); + history_t::add_pending_with_file_detection(history, L"cmd5 ~/" + filename, vars); + history_t::add_pending_with_file_detection(history, L"cmd6 ~/notafile", vars); + history_t::add_pending_with_file_detection(history, L"cmd7 ~/*f*", vars); + history_t::add_pending_with_file_detection(history, L"cmd8 ~/*zzz*", vars); + history->resolve_pending(); constexpr size_t hist_size = 9; - if (history.size() != hist_size) { - err(L"history has wrong size: %lu but expected %lu", (unsigned long)history.size(), (unsigned long)hist_size); - history.clear(); + if (history->size() != hist_size) { + err(L"history has wrong size: %lu but expected %lu", (unsigned long)history->size(), + (unsigned long)hist_size); + history->clear(); return; } @@ -4395,7 +4397,7 @@ void history_tests_t::test_history_path_detection() { int failures = 0; bool last = (lap + 1 == maxlap); for (size_t i = 1; i <= hist_size; i++) { - if (history.item_at_index(i).required_paths != expected[hist_size - i]) { + if (history->item_at_index(i).required_paths != expected[hist_size - i]) { failures += 1; if (last) { err(L"Wrong detected paths for item %lu", (unsigned long)i); @@ -4410,7 +4412,7 @@ void history_tests_t::test_history_path_detection() { usleep(1E6 / 500); // 1 msec } //fprintf(stderr, "History saving took %lu laps\n", (unsigned long)lap); - history.clear(); + history->clear(); } static bool install_sample_history(const wchar_t *name) { @@ -4429,7 +4431,7 @@ static bool install_sample_history(const wchar_t *name) { } /// Indicates whether the history is equal to the given null-terminated array of strings. -static bool history_equals(history_t &hist, const wchar_t *const *strings) { +static bool history_equals(const shared_ptr &hist, const wchar_t *const *strings) { // Count our expected items. size_t expected_count = 0; while (strings[expected_count]) { @@ -4441,7 +4443,7 @@ static bool history_equals(history_t &hist, const wchar_t *const *strings) { size_t array_idx = 0; for (;;) { const wchar_t *expected = strings[array_idx]; - history_item_t item = hist.item_at_index(history_idx); + history_item_t item = hist->item_at_index(history_idx); if (expected == NULL) { if (!item.empty()) { err(L"Expected empty item at history index %lu, instead found: %ls", history_idx, item.str().c_str()); @@ -4473,11 +4475,11 @@ void history_tests_t::test_history_formats() { const wchar_t *const expected[] = { L"#def", L"echo #abc", L"function yay\necho hi\nend", L"cd foobar", L"ls /", NULL}; - history_t &test_history = history_t::history_with_name(name); + auto test_history = history_t::with_name(name); if (!history_equals(test_history, expected)) { err(L"test_history_formats failed for %ls\n", name); } - test_history.clear(); + test_history->clear(); } name = L"history_sample_fish_2_0"; @@ -4488,11 +4490,11 @@ void history_tests_t::test_history_formats() { const wchar_t *const expected[] = {L"echo this has\\\nbackslashes", L"function foo\necho bar\nend", L"echo alpha", NULL}; - history_t &test_history = history_t::history_with_name(name); + auto test_history = history_t::with_name(name); if (!history_equals(test_history, expected)) { err(L"test_history_formats failed for %ls\n", name); } - test_history.clear(); + test_history->clear(); } say(L"Testing bash import"); @@ -4506,12 +4508,12 @@ void history_tests_t::test_history_formats() { L"/** # see issue 7407", L"sleep 123", L"a && echo valid construct", L"final line", L"echo supsup", L"export XVAR='exported'", L"history --help", L"echo foo", NULL}; - history_t &test_history = history_t::history_with_name(L"bash_import"); - test_history.populate_from_bash(f); + auto test_history = history_t::with_name(L"bash_import"); + test_history->populate_from_bash(f); if (!history_equals(test_history, expected)) { err(L"test_history_formats failed for bash import\n"); } - test_history.clear(); + test_history->clear(); fclose(f); } @@ -4521,13 +4523,13 @@ void history_tests_t::test_history_formats() { err(L"Couldn't open file tests/%ls", name); } else { // We simply invoke get_string_representation. If we don't die, the test is a success. - history_t &test_history = history_t::history_with_name(name); + auto test_history = history_t::with_name(name); const wchar_t *expected[] = {L"no_newline_at_end_of_file", L"corrupt_prefix", L"this_command_is_ok", NULL}; if (!history_equals(test_history, expected)) { err(L"test_history_formats failed for %ls\n", name); } - test_history.clear(); + test_history->clear(); } } diff --git a/src/history.cpp b/src/history.cpp index 830d1a2d4..2b6ac98a9 100644 --- a/src/history.cpp +++ b/src/history.cpp @@ -1324,9 +1324,13 @@ void history_t::remove(const wcstring &str) { impl()->remove(str); } void history_t::remove_ephemeral_items() { impl()->remove_ephemeral_items(); } -void history_t::add_pending_with_file_detection(const wcstring &str, +// static +void history_t::add_pending_with_file_detection(const std::shared_ptr &self, + const wcstring &str, const std::shared_ptr &vars, history_persistence_mode_t persist_mode) { + assert(self && "Null history"); + // We use empty items as sentinels to indicate the end of history. // Do not allow them to be added (#6032). if (str.empty()) { @@ -1367,7 +1371,7 @@ void history_t::add_pending_with_file_detection(const wcstring &str, // If we got a path, we'll perform file detection for autosuggestion hinting. bool wants_file_detection = !potential_paths.empty() && !needs_sync_write; - auto imp = this->impl(); + auto imp = self->impl(); // Make our history item. time_t when = imp->timestamp_now(); @@ -1384,7 +1388,7 @@ void history_t::add_pending_with_file_detection(const wcstring &str, iothread_perform([=]() { // Don't hold the lock while we perform this file detection. auto validated_paths = expand_and_detect_paths(potential_paths, *vars); - auto imp = this->impl(); + auto imp = self->impl(); imp->set_valid_file_paths(std::move(validated_paths), identifier); imp->enable_automatic_saving(); }); @@ -1404,7 +1408,7 @@ void history_t::save() { impl()->save(); } /// Perform a search of \p hist for \p search_string. Invoke a function \p func for each match. If /// \p func returns true, continue the search; else stop it. -static void do_1_history_search(history_t &hist, history_search_type_t search_type, +static void do_1_history_search(history_t *hist, history_search_type_t search_type, const wcstring &search_string, bool case_sensitive, const std::function &func, const cancel_checker_t &cancel_check) { @@ -1443,7 +1447,7 @@ bool history_t::search(history_search_type_t search_type, const wcstring_list_t if (search_args.empty()) { // The user had no search terms; just append everything. - do_1_history_search(*this, history_search_type_t::match_everything, {}, false, func, + do_1_history_search(this, history_search_type_t::match_everything, {}, false, func, cancel_check); } else { for (const wcstring &search_string : search_args) { @@ -1451,7 +1455,7 @@ bool history_t::search(history_search_type_t search_type, const wcstring_list_t streams.err.append_format(L"Searching for the empty string isn't allowed"); return false; } - do_1_history_search(*this, search_type, search_string, case_sensitive, func, + do_1_history_search(this, search_type, search_string, case_sensitive, func, cancel_check); } } @@ -1482,7 +1486,7 @@ history_item_t history_t::item_at_index(size_t idx) { return impl()->item_at_ind size_t history_t::size() { return impl()->size(); } /// The set of all histories. -static owning_lock>> s_histories; +static owning_lock>> s_histories; void history_save_all() { auto histories = s_histories.acquire(); @@ -1491,16 +1495,13 @@ void history_save_all() { } } -history_t &history_t::history_with_name(const wcstring &name) { - // Return a history for the given name, creating it if necessary - // Note that histories are currently never deleted, so we can return a reference to them without - // using something like shared_ptr +std::shared_ptr history_t::with_name(const wcstring &name) { auto hs = s_histories.acquire(); - std::unique_ptr &hist = (*hs)[name]; + std::shared_ptr &hist = (*hs)[name]; if (!hist) { - hist = make_unique(name); + hist = std::make_shared(name); } - return *hist; + return hist; } void start_private_mode(env_stack_t &vars) { diff --git a/src/history.h b/src/history.h index 9f5e24c6c..daa22f037 100644 --- a/src/history.h +++ b/src/history.h @@ -163,7 +163,7 @@ class history_t { static bool never_mmap; // Returns history with the given name, creating it if necessary. - static history_t &history_with_name(const wcstring &name); + static std::shared_ptr with_name(const wcstring &name); /// Returns whether this is using the default name. bool is_default() const; @@ -180,9 +180,10 @@ class history_t { // Add a new pending history item to the end, and then begin file detection on the items to // determine which arguments are paths. Arguments may be expanded (e.g. with PWD and variables) - // using the given \p vars. The item has the given \p persist_mode - void add_pending_with_file_detection( - const wcstring &str, const std::shared_ptr &vars, + // using the given \p vars. The item has the given \p persist_mode. + static void add_pending_with_file_detection( + const std::shared_ptr &self, const wcstring &str, + const std::shared_ptr &vars, history_persistence_mode_t persist_mode = history_persistence_mode_t::disk); // Resolves any pending history items, so that they may be returned in history searches. @@ -241,6 +242,7 @@ using history_search_flags_t = uint32_t; class history_search_t { private: // The history in which we are searching. + // TODO: this should be a shared_ptr. history_t *history_; // The original search term. @@ -283,16 +285,23 @@ class history_search_t { // Returns the current search result item contents. asserts if there is no current item. const wcstring ¤t_string() const; - // Constructor. - history_search_t(history_t &hist, const wcstring &str, + // Construct from a history pointer; the caller is responsible for ensuring the history stays + // alive. + history_search_t(history_t *hist, const wcstring &str, enum history_search_type_t type = history_search_type_t::contains, history_search_flags_t flags = 0) - : history_(&hist), orig_term_(str), canon_term_(str), search_type_(type), flags_(flags) { + : history_(hist), orig_term_(str), canon_term_(str), search_type_(type), flags_(flags) { if (ignores_case()) { std::transform(canon_term_.begin(), canon_term_.end(), canon_term_.begin(), towlower); } } + // Construct from a shared_ptr. TODO: this should be the only constructor. + history_search_t(const std::shared_ptr &hist, const wcstring &str, + enum history_search_type_t type = history_search_type_t::contains, + history_search_flags_t flags = 0) + : history_search_t(hist.get(), str, type, flags) {} + // Default constructor. history_search_t() = default; }; diff --git a/src/reader.cpp b/src/reader.cpp index 3b3f1a514..357c0fc17 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -446,7 +446,7 @@ class reader_history_search_t { bool add_skip(const wcstring &str) { return skips_.insert(str).second; } /// Reset, beginning a new line or token mode search. - void reset_to_mode(const wcstring &text, history_t *hist, mode_t mode) { + void reset_to_mode(const wcstring &text, const std::shared_ptr &hist, mode_t mode) { assert(mode != inactive && "mode cannot be inactive in this setter"); skips_ = {text}; matches_ = {text}; @@ -458,7 +458,7 @@ class reader_history_search_t { if (low == text) flags |= history_search_ignore_case; // We can skip dedup in history_search_t because we do it ourselves in skips_. search_ = history_search_t( - *hist, text, + hist, text, by_prefix() ? history_search_type_t::prefix : history_search_type_t::contains, flags); } @@ -586,7 +586,7 @@ class reader_data_t : public std::enable_shared_from_this { /// The source of input events. inputter_t inputter; /// The history. - history_t *history{nullptr}; + std::shared_ptr history{}; /// The history search. reader_history_search_t history_search{}; @@ -680,11 +680,12 @@ class reader_data_t : public std::enable_shared_from_this { /// Access the parser. parser_t &parser() { return *parser_ref; } - reader_data_t(std::shared_ptr parser, history_t *hist, reader_config_t &&conf) + reader_data_t(std::shared_ptr parser, std::shared_ptr hist, + reader_config_t &&conf) : conf(std::move(conf)), parser_ref(std::move(parser)), inputter(*parser_ref, conf.in), - history(hist) {} + history(std::move(hist)) {} void update_buff_pos(editable_line_t *el, maybe_t new_pos = none_t()); @@ -1630,7 +1631,8 @@ void reader_data_t::completion_insert(const wcstring &val, size_t token_end, // Returns a function that can be invoked (potentially // on a background thread) to determine the autosuggestion static std::function get_autosuggestion_performer( - parser_t &parser, const wcstring &search_string, size_t cursor_pos, history_t *history) { + parser_t &parser, const wcstring &search_string, size_t cursor_pos, + const std::shared_ptr &history) { const uint32_t generation_count = read_generation_count(); auto vars = parser.vars().snapshot(); const wcstring working_directory = vars->get_pwd_slash(); @@ -1651,7 +1653,7 @@ static std::function get_autosuggestion_performer( } // Search history for a matching item. - history_search_t searcher(*history, search_string, history_search_type_t::prefix, + history_search_t searcher(history.get(), search_string, history_search_type_t::prefix, history_search_flags_t{}); while (!ctx.check_cancel() && searcher.go_backwards()) { const history_item_t &item = searcher.current_item(); @@ -2512,7 +2514,7 @@ void reader_change_history(const wcstring &name) { reader_data_t *data = current_data_or_null(); if (data && data->history) { data->history->save(); - data->history = &history_t::history_with_name(name); + data->history = history_t::with_name(name); } } @@ -2521,7 +2523,7 @@ void reader_change_history(const wcstring &name) { static std::shared_ptr reader_push_ret(parser_t &parser, const wcstring &history_name, reader_config_t &&conf) { - history_t *hist = &history_t::history_with_name(history_name); + std::shared_ptr hist = history_t::with_name(history_name); auto data = std::make_shared(parser.shared(), hist, std::move(conf)); reader_data_stack.push_back(data); data->command_line_changed(&data->command_line); @@ -3241,7 +3243,8 @@ void reader_data_t::handle_readline_command(readline_cmd_t c, readline_loop_stat } else { mode = history_persistence_mode_t::disk; } - history->add_pending_with_file_detection(text, vars.snapshot(), mode); + history_t::add_pending_with_file_detection(history, text, vars.snapshot(), + mode); } rls.finished = true; @@ -4061,7 +4064,7 @@ const wchar_t *reader_get_buffer() { return data ? data->command_line.text().c_str() : nullptr; } -history_t *reader_get_history() { +std::shared_ptr reader_get_history() { ASSERT_IS_MAIN_THREAD(); reader_data_t *data = current_data_or_null(); return data ? data->history : nullptr; diff --git a/src/reader.h b/src/reader.h index 7acb76c3a..b7e7a3e15 100644 --- a/src/reader.h +++ b/src/reader.h @@ -169,7 +169,7 @@ void reader_queue_ch(const char_event_t &ch); const wchar_t *reader_get_buffer(); /// Returns the current reader's history. -history_t *reader_get_history(); +std::shared_ptr reader_get_history(); /// Set the string of characters in the command buffer, as well as the cursor position. /// From f496b07c7cc9ebf92a64762876a746ee386c0f02 Mon Sep 17 00:00:00 2001 From: Collin Styles Date: Sat, 9 Jan 2021 23:54:45 -0800 Subject: [PATCH 092/105] Fix completion for `--exact` option to `fzf` These double hyphens will make the completion resolve to `----exact` which isn't a valid option. --- share/completions/fzf.fish | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/share/completions/fzf.fish b/share/completions/fzf.fish index ccc2e0126..cd2d2a82f 100644 --- a/share/completions/fzf.fish +++ b/share/completions/fzf.fish @@ -3,7 +3,7 @@ complete -c fzf -f # Search mode complete -c fzf -l no-extended -d no-extended complete -c fzf -n 'string match "+*" -- (commandline -ct)' -a +x -d no-extended -complete -c fzf -s e -l --exact -d 'Enable exact-match' +complete -c fzf -s e -l exact -d 'Enable exact-match' complete -c fzf -n 'string match "+*" -- (commandline -ct)' -a +i -d 'case-sensitive match' complete -c fzf -s i -d 'Case-insensitive match' complete -c fzf -l literal -d 'Do not normalize latin script letters for matching' From bb3b6e3329504d087215d109168c4fa2affb4aaa Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Sun, 10 Jan 2021 18:33:31 +0100 Subject: [PATCH 093/105] completions/timedatectl: Add missing quotes Oops! Supersedes #7617. --- share/completions/timedatectl.fish | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/share/completions/timedatectl.fish b/share/completions/timedatectl.fish index e99212075..8c3e1ae50 100644 --- a/share/completions/timedatectl.fish +++ b/share/completions/timedatectl.fish @@ -5,7 +5,7 @@ complete -c timedatectl -n "not __fish_seen_subcommand_from $commands" -a status complete -c timedatectl -n "not __fish_seen_subcommand_from $commands" -a show -d 'Show properties of systemd-timedated' complete -c timedatectl -n "not __fish_seen_subcommand_from $commands" -a set-time -d 'Set system time' complete -c timedatectl -n "not __fish_seen_subcommand_from $commands" -a set-timezone -d 'Set system time zone' -complete -c timedatectl -n "__fish_seen_subcommand_from set-timezone" -a (timedatectl list-timezones) +complete -c timedatectl -n "__fish_seen_subcommand_from set-timezone" -a "(timedatectl list-timezones)" complete -c timedatectl -n "not __fish_seen_subcommand_from $commands" -a list-timezones -d 'Show known time zones' complete -c timedatectl -n "not __fish_seen_subcommand_from $commands" -a set-local-rtc -d 'Control whether RTC is in local time' complete -c timedatectl -n "__fish_seen_subcommand_from set-local-rtc" -a 'true false' From 20d91c6be2e32390dae3a95b3ec04f1c1397ecf7 Mon Sep 17 00:00:00 2001 From: exploide Date: Sun, 10 Jan 2021 16:32:44 +0100 Subject: [PATCH 094/105] added completion script for alternatives --- share/completions/alternatives.fish | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 share/completions/alternatives.fish diff --git a/share/completions/alternatives.fish b/share/completions/alternatives.fish new file mode 100644 index 000000000..1b7e1af68 --- /dev/null +++ b/share/completions/alternatives.fish @@ -0,0 +1,29 @@ +function __fish_print_alternatives_names -d "Get the names of link groups in the alternatives system" + alternatives --list | cut -f 1 | string trim +end + +# common options + +complete -c alternatives -l verbose -d "Generate more comments about what alternatives is doing" +complete -c alternatives -l help -d "Give some usage information" +complete -c alternatives -l version -d "Tell which version of alternatives this is" +complete -c alternatives -l keep-missing -d "If new variant doesn't provide some files, keep previous links" +complete -c alternatives -l altdir -xa "(__fish_complete_directories)" -d "Specifies the alternatives directory" +complete -c alternatives -l admindir -xa "(__fish_complete_directories)" -d "Specifies the administrative directory" + +# actions + +complete -c alternatives -l install -r -d "Add a group of alternatives to the system" +complete -c alternatives -l slave -n "__fish_contains_opt install" -r -d "Add a slave link to the new group" +complete -c alternatives -l initscript -n "__fish_contains_opt install" -F -d "Add an initscript for the new group" +complete -c alternatives -l family -n "__fish_contains_opt install" -x -d "Set a family for the new group" + +complete -c alternatives -l remove -ra "(__fish_print_alternatives_names)" -d "Remove an alternative and all of its associated slave links" +complete -c alternatives -l set -ra "(__fish_print_alternatives_names)" -d "Set link group to given path" +complete -c alternatives -l config -xa "(__fish_print_alternatives_names)" -d "Open menu to configure link group" +complete -c alternatives -l auto -xa "(__fish_print_alternatives_names)" -d "Switch the master symlink name to automatic mode" +complete -c alternatives -l display -xa "(__fish_print_alternatives_names)" -d "Display information about the link group" +complete -c alternatives -l list -f -d "Display information about all link groups" +complete -c alternatives -l remove-all -xa "(__fish_print_alternatives_names)" -d "Remove the whole link group name" +complete -c alternatives -l add-slave -ra "(__fish_print_alternatives_names)" -d "Add a slave link to an existing alternative" +complete -c alternatives -l remove-slave -ra "(__fish_print_alternatives_names)" -d "Remove slave from an existing alternative" From 19efd22468a9a7fe1fea49000e2aa956972b8b0b Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Mon, 11 Jan 2021 18:49:15 +0100 Subject: [PATCH 095/105] env: Setup $HOME/$USER *before* the config directories They are based on $HOME, so setting $HOME has to be done first. Fixes #7620 (untested because I'm assuming common CI systems have weird $HOME settings) --- CHANGELOG.rst | 1 + src/env.cpp | 82 ++++++++++++++++++++++++++------------------------- 2 files changed, 43 insertions(+), 40 deletions(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 463f5e0e9..865228538 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -112,6 +112,7 @@ Scripting improvements - ``functions`` now explains when a function was defined via ``source`` instead of just saying ``Defined in -``. - Significant performance improvements when globbing or in ``math``. - ``echo`` no longer interprets options at the beginning of an argument (``echo "-n foo"``) (:issue:`7614`). +- Fish now better handles an unset $HOME (:issue:`7620`). Interactive improvements ------------------------ diff --git a/src/env.cpp b/src/env.cpp index 99fd3c7ca..ff13b6273 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -273,17 +273,6 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { vars.set_one(FISH_BIN_DIR, ENV_GLOBAL, paths->bin); } - wcstring user_config_dir; - path_get_config(user_config_dir); - vars.set_one(FISH_CONFIG_DIR, ENV_GLOBAL, user_config_dir); - - wcstring user_data_dir; - path_get_data(user_data_dir); - vars.set_one(FISH_USER_DATA_DIR, ENV_GLOBAL, user_data_dir); - - // Set up the USER and PATH variables - setup_path(); - // Some `su`s keep $USER when changing to root. // This leads to issues later on (and e.g. in prompts), // so we work around it by resetting $USER. @@ -292,40 +281,13 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { uid_t uid = getuid(); setup_user(uid == 0); - // Set up $IFS - this used to be in share/config.fish, but really breaks if it isn't done. - vars.set_one(L"IFS", ENV_GLOBAL, L"\n \t"); - - // Set up the version variable. - wcstring version = str2wcstring(get_fish_version()); - vars.set_one(L"version", ENV_GLOBAL, version); - vars.set_one(L"FISH_VERSION", ENV_GLOBAL, version); - - // Set the $fish_pid variable. - vars.set_one(L"fish_pid", ENV_GLOBAL, to_string(getpid())); - - // Set the $hostname variable - wcstring hostname = L"fish"; - get_hostname_identifier(hostname); - vars.set_one(L"hostname", ENV_GLOBAL, hostname); - - // Set up SHLVL variable. Not we can't use vars.get() because SHLVL is read-only, and therefore - // was not inherited from the environment. - wcstring nshlvl_str = L"1"; - if (const char *shlvl_var = getenv("SHLVL")) { - const wchar_t *end; - // TODO: Figure out how to handle invalid numbers better. Shouldn't we issue a diagnostic? - long shlvl_i = fish_wcstol(str2wcstring(shlvl_var).c_str(), &end); - if (!errno && shlvl_i >= 0) { - nshlvl_str = to_string(shlvl_i + 1); - } - } - vars.set_one(L"SHLVL", ENV_GLOBAL | ENV_EXPORT, nshlvl_str); - // Set up the HOME variable. // Unlike $USER, it doesn't seem that `su`s pass this along // if the target user is root, unless "--preserve-environment" is used. // Since that is an explicit choice, we should allow it to enable e.g. // env HOME=(mktemp -d) su --preserve-environment fish + // + // Note: This needs to be *before* path_get_*, because that uses $HOME! if (vars.get(L"HOME").missing_or_empty()) { auto user_var = vars.get(L"USER"); if (!user_var.missing_or_empty()) { @@ -357,6 +319,46 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { } } + wcstring user_config_dir; + path_get_config(user_config_dir); + vars.set_one(FISH_CONFIG_DIR, ENV_GLOBAL, user_config_dir); + + wcstring user_data_dir; + path_get_data(user_data_dir); + vars.set_one(FISH_USER_DATA_DIR, ENV_GLOBAL, user_data_dir); + + // Set up a default PATH + setup_path(); + + // Set up $IFS - this used to be in share/config.fish, but really breaks if it isn't done. + vars.set_one(L"IFS", ENV_GLOBAL, L"\n \t"); + + // Set up the version variable. + wcstring version = str2wcstring(get_fish_version()); + vars.set_one(L"version", ENV_GLOBAL, version); + vars.set_one(L"FISH_VERSION", ENV_GLOBAL, version); + + // Set the $fish_pid variable. + vars.set_one(L"fish_pid", ENV_GLOBAL, to_string(getpid())); + + // Set the $hostname variable + wcstring hostname = L"fish"; + get_hostname_identifier(hostname); + vars.set_one(L"hostname", ENV_GLOBAL, hostname); + + // Set up SHLVL variable. Not we can't use vars.get() because SHLVL is read-only, and therefore + // was not inherited from the environment. + wcstring nshlvl_str = L"1"; + if (const char *shlvl_var = getenv("SHLVL")) { + const wchar_t *end; + // TODO: Figure out how to handle invalid numbers better. Shouldn't we issue a diagnostic? + long shlvl_i = fish_wcstol(str2wcstring(shlvl_var).c_str(), &end); + if (!errno && shlvl_i >= 0) { + nshlvl_str = to_string(shlvl_i + 1); + } + } + vars.set_one(L"SHLVL", ENV_GLOBAL | ENV_EXPORT, nshlvl_str); + // initialize the PWD variable if necessary // Note we may inherit a virtual PWD that doesn't match what getcwd would return; respect that // if and only if it matches getcwd (#5647). Note we treat PWD as read-only so it was not set in From f7b2bf8229294e52a4f7276eb9a0e75321e5d812 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Mon, 11 Jan 2021 20:53:11 +0100 Subject: [PATCH 096/105] output: Simplify some duplicated code Becomes a bit boring after a while --- src/output.cpp | 95 +++++++++----------------------------------------- src/output.h | 8 +++++ 2 files changed, 25 insertions(+), 78 deletions(-) diff --git a/src/output.cpp b/src/output.cpp index 5eb76c199..05b3bd683 100644 --- a/src/output.cpp +++ b/src/output.cpp @@ -147,44 +147,21 @@ bool outputter_t::write_color(rgb_color_t color, bool is_fg) { /// \param c Foreground color. /// \param c2 Background color. void outputter_t::set_color(rgb_color_t c, rgb_color_t c2) { - if (!cur_term) return; + // Test if we have at least basic support for setting fonts, colors and related bits - otherwise + // just give up... + if (!cur_term || !exit_attribute_mode) return; const rgb_color_t normal = rgb_color_t::normal(); bool bg_set = false, last_bg_set = false; - bool is_bold = false; - bool is_underline = false; - bool is_italics = false; - bool is_dim = false; - bool is_reverse = false; - - // Test if we have at least basic support for setting fonts, colors and related bits - otherwise - // just give up... - if (!exit_attribute_mode) { - return; - } - - is_bold |= c.is_bold(); - is_bold |= c2.is_bold(); - - is_underline |= c.is_underline(); - is_underline |= c2.is_underline(); - - is_italics |= c.is_italics(); - is_italics |= c2.is_italics(); - - is_dim |= c.is_dim(); - is_dim |= c2.is_dim(); - - is_reverse |= c.is_reverse(); - is_reverse |= c2.is_reverse(); + bool is_bold = c.is_bold() || c2.is_bold(); + bool is_underline = c.is_underline() || c2.is_underline(); + bool is_italics = c.is_italics() || c2.is_italics(); + bool is_dim = c.is_dim() || c2.is_dim(); + bool is_reverse = c.is_reverse() || c2.is_reverse(); if (c.is_reset() || c2.is_reset()) { c = c2 = normal; - was_bold = false; - was_underline = false; - was_italics = false; - was_dim = false; - was_reverse = false; + reset_modes(); // If we exit attibute mode, we must first set a color, or previously colored text might // lose it's color. Terminals are weird... write_foreground_color(*this, 0); @@ -192,40 +169,14 @@ void outputter_t::set_color(rgb_color_t c, rgb_color_t c2) { return; } - if (was_bold && !is_bold) { - // Only way to exit bold mode is a reset of all attributes. + if ((was_bold && !is_bold) + || (was_dim && !is_dim) + || (was_reverse && !is_reverse)) { + // Only way to exit bold/dim/reverse mode is a reset of all attributes. writembs(*this, exit_attribute_mode); last_color = normal; last_color2 = normal; - was_bold = false; - was_underline = false; - was_italics = false; - was_dim = false; - was_reverse = false; - } - - if (was_dim && !is_dim) { - // Only way to exit dim mode is a reset of all attributes. - writembs(*this, exit_attribute_mode); - last_color = normal; - last_color2 = normal; - was_bold = false; - was_underline = false; - was_italics = false; - was_dim = false; - was_reverse = false; - } - - if (was_reverse && !is_reverse) { - // Only way to exit reverse mode is a reset of all attributes. - writembs(*this, exit_attribute_mode); - last_color = normal; - last_color2 = normal; - was_bold = false; - was_underline = false; - was_italics = false; - was_dim = false; - was_reverse = false; + reset_modes(); } if (!last_color2.is_normal() && !last_color2.is_reset()) { @@ -248,11 +199,7 @@ void outputter_t::set_color(rgb_color_t c, rgb_color_t c2) { if (!bg_set && last_bg_set) { // Background color changed and is no longer set, so we exit bold mode. writembs(*this, exit_attribute_mode); - was_bold = false; - was_underline = false; - was_italics = false; - was_dim = false; - was_reverse = false; + reset_modes(); // We don't know if exit_attribute_mode resets colors, so we set it to something known. if (write_foreground_color(*this, 0)) { last_color = rgb_color_t::black(); @@ -266,11 +213,7 @@ void outputter_t::set_color(rgb_color_t c, rgb_color_t c2) { writembs(*this, exit_attribute_mode); last_color2 = rgb_color_t::normal(); - was_bold = false; - was_underline = false; - was_italics = false; - was_dim = false; - was_reverse = false; + reset_modes(); } else if (!c.is_special()) { write_color(c, true /* foreground */); } @@ -287,11 +230,7 @@ void outputter_t::set_color(rgb_color_t c, rgb_color_t c2) { write_color(last_color, true /* foreground */); } - was_bold = false; - was_underline = false; - was_italics = false; - was_dim = false; - was_reverse = false; + reset_modes(); last_color2 = c2; } else if (!c2.is_special()) { write_color(c2, false /* not foreground */); diff --git a/src/output.h b/src/output.h index 326364d9c..e3d2d1c11 100644 --- a/src/output.h +++ b/src/output.h @@ -32,6 +32,14 @@ class outputter_t { bool was_dim = false; bool was_reverse = false; + void reset_modes() { + was_bold = false; + was_underline = false; + was_italics = false; + was_dim = false; + was_reverse = false; + } + /// Construct an outputter which outputs to a given fd. explicit outputter_t(int fd) : fd_(fd) {} From 7bf2b9fd43c444b74983e31483493feb3c2ebf58 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Mon, 11 Jan 2021 20:56:15 +0100 Subject: [PATCH 097/105] output: Rename some variables These are a foreground and a background color. Now I see the point in not naming them "foreground_color" and "background_color", but at least "fg" and "bg" should do, right? --- src/output.cpp | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/src/output.cpp b/src/output.cpp index 05b3bd683..2dfe1566b 100644 --- a/src/output.cpp +++ b/src/output.cpp @@ -144,23 +144,23 @@ bool outputter_t::write_color(rgb_color_t color, bool is_fg) { /// - Lastly we may need to write set_a_background or set_a_foreground to set the other half of the /// color pair to what it should be. /// -/// \param c Foreground color. -/// \param c2 Background color. -void outputter_t::set_color(rgb_color_t c, rgb_color_t c2) { +/// \param fg Foreground color. +/// \param bg Background color. +void outputter_t::set_color(rgb_color_t fg, rgb_color_t bg) { // Test if we have at least basic support for setting fonts, colors and related bits - otherwise // just give up... if (!cur_term || !exit_attribute_mode) return; const rgb_color_t normal = rgb_color_t::normal(); bool bg_set = false, last_bg_set = false; - bool is_bold = c.is_bold() || c2.is_bold(); - bool is_underline = c.is_underline() || c2.is_underline(); - bool is_italics = c.is_italics() || c2.is_italics(); - bool is_dim = c.is_dim() || c2.is_dim(); - bool is_reverse = c.is_reverse() || c2.is_reverse(); + bool is_bold = fg.is_bold() || bg.is_bold(); + bool is_underline = fg.is_underline() || bg.is_underline(); + bool is_italics = fg.is_italics() || bg.is_italics(); + bool is_dim = fg.is_dim() || bg.is_dim(); + bool is_reverse = fg.is_reverse() || bg.is_reverse(); - if (c.is_reset() || c2.is_reset()) { - c = c2 = normal; + if (fg.is_reset() || bg.is_reset()) { + fg = bg = normal; reset_modes(); // If we exit attibute mode, we must first set a color, or previously colored text might // lose it's color. Terminals are weird... @@ -184,10 +184,10 @@ void outputter_t::set_color(rgb_color_t c, rgb_color_t c2) { last_bg_set = true; } - if (!c2.is_normal()) { + if (!bg.is_normal()) { // Background is set. bg_set = true; - if (c == c2) c = (c2 == rgb_color_t::white()) ? rgb_color_t::black() : rgb_color_t::white(); + if (fg == bg) fg = (bg == rgb_color_t::white()) ? rgb_color_t::black() : rgb_color_t::white(); } if (enter_bold_mode && enter_bold_mode[0] != '\0') { @@ -207,22 +207,22 @@ void outputter_t::set_color(rgb_color_t c, rgb_color_t c2) { } } - if (last_color != c) { - if (c.is_normal()) { + if (last_color != fg) { + if (fg.is_normal()) { write_foreground_color(*this, 0); writembs(*this, exit_attribute_mode); last_color2 = rgb_color_t::normal(); reset_modes(); - } else if (!c.is_special()) { - write_color(c, true /* foreground */); + } else if (!fg.is_special()) { + write_color(fg, true /* foreground */); } } - last_color = c; + last_color = fg; - if (last_color2 != c2) { - if (c2.is_normal()) { + if (last_color2 != bg) { + if (bg.is_normal()) { write_background_color(*this, 0); writembs(*this, exit_attribute_mode); @@ -231,10 +231,10 @@ void outputter_t::set_color(rgb_color_t c, rgb_color_t c2) { } reset_modes(); - last_color2 = c2; - } else if (!c2.is_special()) { - write_color(c2, false /* not foreground */); - last_color2 = c2; + last_color2 = bg; + } else if (!bg.is_special()) { + write_color(bg, false /* not foreground */); + last_color2 = bg; } } From 3fc9c0b38c7141532f278b0f46cebe25232fc166 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Mon, 11 Jan 2021 21:00:33 +0100 Subject: [PATCH 098/105] tests: Increase cancellation delay This sometimes fails on github actions with ASAN. I am assuming that's because the ctrl-c happens *before* the process has had a chance to start. So we do what we do and increase the delay. --- src/fish_tests.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 0331cf77c..9ba875b35 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -1295,7 +1295,7 @@ static void test_parser() { static void test_1_cancellation(const wchar_t *src) { auto filler = io_bufferfill_t::create(fd_set_t{}); pthread_t thread = pthread_self(); - double delay = 0.25 /* seconds */; + double delay = 0.50 /* seconds */; iothread_perform([=]() { /// Wait a while and then SIGINT the main thread. usleep(delay * 1E6); From 32c65aa32cccad8738361c76c84719872092c06f Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Mon, 11 Jan 2021 21:03:10 +0100 Subject: [PATCH 099/105] Lock threads only once a day This ran hourly, and that's really not necessary anymore. --- .github/workflows/lockthreads.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/lockthreads.yml b/.github/workflows/lockthreads.yml index 60699703a..3a520f357 100644 --- a/.github/workflows/lockthreads.yml +++ b/.github/workflows/lockthreads.yml @@ -2,7 +2,7 @@ name: 'Lock threads' on: schedule: - - cron: '0 * * * *' + - cron: '0 18 * * *' jobs: lock: From 2709467b7357c93ae31ce2b8e622ac80180db276 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Mon, 11 Jan 2021 12:39:24 -0800 Subject: [PATCH 100/105] Add a Ubuntu bionic asan clang dockerfile test This may be run with: ./docker/docker_run_tests.sh ./docker/bionic-asan-clang.Dockerfile --- docker/bionic-asan-clang.Dockerfile | 38 +++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 docker/bionic-asan-clang.Dockerfile diff --git a/docker/bionic-asan-clang.Dockerfile b/docker/bionic-asan-clang.Dockerfile new file mode 100644 index 000000000..dfdbd086a --- /dev/null +++ b/docker/bionic-asan-clang.Dockerfile @@ -0,0 +1,38 @@ +FROM ubuntu:18.04 + +ENV LANG C.UTF-8 +ENV LC_ALL C.UTF-8 +ENV CXXFLAGS="-fno-omit-frame-pointer -fsanitize=undefined -fsanitize=address" \ + CC=clang-9 \ + CXX=clang++-9 \ + ASAN_OPTIONS=check_initialization_order=1:detect_stack_use_after_return=1:detect_leaks=1 \ + UBSAN_OPTIONS=print_stacktrace=1:report_error_type=1 + +RUN apt-get update \ + && apt-get -y install \ + build-essential \ + cmake \ + clang-9 \ + gettext \ + git \ + libncurses5-dev \ + locales \ + ninja-build \ + python3 \ + python3-pexpect \ + sudo \ + && locale-gen en_US.UTF-8 + +RUN groupadd -g 1000 fishuser \ + && useradd -p $(openssl passwd -1 fish) -d /home/fishuser -m -u 1000 -g 1000 fishuser \ + && adduser fishuser sudo \ + && mkdir -p /home/fishuser/fish-build \ + && mkdir /fish-source \ + && chown -R fishuser:fishuser /home/fishuser /fish-source + +USER fishuser +WORKDIR /home/fishuser + +COPY fish_run_tests.sh / + +CMD /fish_run_tests.sh From 7207a205f2e19fde30d3926f03d5c5c878048d43 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Sun, 10 Jan 2021 17:08:34 -0800 Subject: [PATCH 101/105] Switch history races test to use threads instead of processes This avoids issues with ASan and TSan whose allocators do not properly clean up in atfork, leading to deadlocks in child processes. --- src/fish_tests.cpp | 53 +++++++++++++++++++++++++--------------------- 1 file changed, 29 insertions(+), 24 deletions(-) diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 9ba875b35..3556cc48f 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -3985,7 +3985,7 @@ class history_tests_t { static void test_history_formats(); // static void test_history_speed(void); static void test_history_races(); - static void test_history_races_pound_on_history(size_t item_count); + static void test_history_races_pound_on_history(size_t item_count, size_t idx); }; static wcstring random_string() { @@ -4113,7 +4113,6 @@ void history_tests_t::test_history() { // Clean up after our tests. history->clear(); } - // Wait until the next second. static void time_barrier() { time_t start = time(NULL); @@ -4122,20 +4121,19 @@ static void time_barrier() { } while (time(NULL) == start); } -static wcstring_list_t generate_history_lines(size_t item_count, int pid) { +static wcstring_list_t generate_history_lines(size_t item_count, size_t idx) { wcstring_list_t result; result.reserve(item_count); for (unsigned long i = 0; i < item_count; i++) { - result.push_back(format_string(L"%ld %lu", (long)pid, (unsigned long)i)); + result.push_back(format_string(L"%ld %lu", (unsigned long)idx, (unsigned long)i)); } return result; } -void history_tests_t::test_history_races_pound_on_history(size_t item_count) { - // Called in child process to modify history. +void history_tests_t::test_history_races_pound_on_history(size_t item_count, size_t idx) { + // Called in child thread to modify history. history_t hist(L"race_test"); - hist.chaos_mode = !true; - const wcstring_list_t hist_lines = generate_history_lines(item_count, getpid()); + const wcstring_list_t hist_lines = generate_history_lines(item_count, idx); for (const wcstring &line : hist_lines) { hist.add(line); hist.save(); @@ -4145,6 +4143,21 @@ void history_tests_t::test_history_races_pound_on_history(size_t item_count) { void history_tests_t::test_history_races() { say(L"Testing history race conditions"); + // It appears TSAN and ASAN's allocators do not release their locks properly in atfork, so + // allocating with multiple threads risks deadlock. Drain threads before running under ASAN. + // TODO: stop forking with these tests. + bool needs_thread_drain = false; +#if __SANITIZE_ADDRESS__ + needs_thread_drain |= true; +#endif +#if defined(__has_feature) + needs_thread_drain |= __has_feature(thread_sanitizer) || __has_feature(address_sanitizer); +#endif + + if (needs_thread_drain) { + iothread_drain_all(); + } + // Test concurrent history writing. // How many concurrent writers we have constexpr size_t RACE_COUNT = 4; @@ -4155,30 +4168,22 @@ void history_tests_t::test_history_races() { // Ensure history is clear. history_t(L"race_test").clear(); - pid_t children[RACE_COUNT]; - for (pid_t &child : children) { - pid_t pid = fork(); - if (!pid) { - // Child process. - setup_fork_guards(); - test_history_races_pound_on_history(ITEM_COUNT); - exit_without_destructors(0); - } else { - // Parent process. - child = pid; - } + // hist.chaos_mode = true; + + std::thread children[RACE_COUNT]; + for (size_t i = 0; i < RACE_COUNT; i++) { + children[i] = std::thread([=] { test_history_races_pound_on_history(ITEM_COUNT, i); }); } // Wait for all children. - for (pid_t child : children) { - int stat; - waitpid(child, &stat, WUNTRACED); + for (std::thread &child : children) { + child.join(); } // Compute the expected lines. std::array expected_lines; for (size_t i = 0; i < RACE_COUNT; i++) { - expected_lines[i] = generate_history_lines(ITEM_COUNT, children[i]); + expected_lines[i] = generate_history_lines(ITEM_COUNT, i); } // Ensure we consider the lines that have been outputted as part of our history. From 1d4883d81043c70e7ca03c0e3b793076c10e4deb Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Mon, 11 Jan 2021 15:23:52 -0800 Subject: [PATCH 102/105] Remove an unnecessary 'using' declaration This was just redundant with the struct tag. --- src/builtin_string.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/builtin_string.cpp b/src/builtin_string.cpp index bc3b0144a..3dce746e4 100644 --- a/src/builtin_string.cpp +++ b/src/builtin_string.cpp @@ -134,7 +134,7 @@ class arg_iterator_t { // This is used by the string subcommands to communicate with the option parser which flags are // valid and get the result of parsing the command for flags. -using options_t = struct options_t { //!OCLINT(too many fields) +struct options_t { //!OCLINT(too many fields) bool all_valid = false; bool char_to_pad_valid = false; bool chars_to_trim_valid = false; From 290d1f2cd6252455ac62b8d16f99a91d26be6624 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Mon, 11 Jan 2021 15:36:54 -0800 Subject: [PATCH 103/105] Mild refactoring of builtin_string repeat Preparation for fixing issue 5988; no behavior change expected here. --- src/builtin_string.cpp | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/src/builtin_string.cpp b/src/builtin_string.cpp index 3dce746e4..d8ae564d5 100644 --- a/src/builtin_string.cpp +++ b/src/builtin_string.cpp @@ -1558,7 +1558,7 @@ static int string_repeat(parser_t &parser, io_streams_t &streams, int argc, wcha int retval = parse_opts(&opts, &optind, 0, argc, argv, parser, streams); if (retval != STATUS_CMD_OK) return retval; - bool is_empty = true; + bool all_empty = true; arg_iterator_t aiter(argv, optind, streams); if (const wcstring *word = aiter.nextstr()) { @@ -1567,17 +1567,24 @@ static int string_repeat(parser_t &parser, io_streams_t &streams, int argc, wcha !opts.count; const wcstring repeated = limit_repeat ? wcsrepeat_until(*word, opts.max) : wcsrepeat(*word, opts.count); - is_empty = repeated.empty(); - - if (!opts.quiet && !is_empty) { + if (!repeated.empty()) { + all_empty = false; + if (opts.quiet) { + // Early out if we can - see #7495. + return STATUS_CMD_OK; + } + } + if (!opts.quiet) { streams.out.append(repeated); - if (!opts.no_newline) streams.out.append(L"\n"); - } else if (opts.quiet && !is_empty) { - return STATUS_CMD_OK; } } - return !is_empty ? STATUS_CMD_OK : STATUS_CMD_ERROR; + // Historical behavior is to never append a newline if all strings were empty. + if (!opts.quiet && !opts.no_newline && !all_empty) { + streams.out.append(L'\n'); + } + + return all_empty ? STATUS_CMD_ERROR : STATUS_CMD_OK; } static int string_sub(parser_t &parser, io_streams_t &streams, int argc, wchar_t **argv) { From 7a0bddfcfa60a4cc5b57792bd1fa88168fec0ce4 Mon Sep 17 00:00:00 2001 From: ridiculousfish Date: Mon, 11 Jan 2021 16:30:33 -0800 Subject: [PATCH 104/105] Teach string repeat to handle multiple arguments Each argument in string repeat is handled independently, except that the --no-newline option applies only to the last newline. Fixes #5988 --- CHANGELOG.rst | 1 + src/builtin_string.cpp | 9 ++++++++- tests/checks/string.fish | 21 +++++++++++++++++++++ 3 files changed, 30 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 865228538..e44e651d7 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -103,6 +103,7 @@ Scripting improvements - ``math`` learned bitwise functions ``bitand``, ``bitor`` and ``bitxor``, used like ``math "bitand(0xFE, 5)"`` (:issue:`7281`). - ``math`` learned tau for those wishing to cut down on typing "2 * pi". - ``string`` subcommands now quit early when used with ``--quiet`` (:issue:`7495`). +- ``string repeat`` now handles multiple arguments, repeating each one (:issue:`5988`). - Failed redirections will now set ``$status`` (:issue:`7540`). - More consistent $status after errors, including invalid expansions like ``$foo[``. - ``read`` can now read interactively from other files, so e.g. forcing it to read from the terminal via ``read 0 && word->length() * opts.count > static_cast(opts.max)) || !opts.count; @@ -1574,6 +1579,8 @@ static int string_repeat(parser_t &parser, io_streams_t &streams, int argc, wcha return STATUS_CMD_OK; } } + + // Append if not quiet. if (!opts.quiet) { streams.out.append(repeated); } diff --git a/tests/checks/string.fish b/tests/checks/string.fish index 60eab3d89..cd2d03626 100644 --- a/tests/checks/string.fish +++ b/tests/checks/string.fish @@ -411,6 +411,27 @@ string repeat -n3 -m20 foo string repeat -m4 foo # CHECK: foof +string repeat -n 5 a b c +# CHECK: aaaaa +# CHECK: bbbbb +# CHECK: ccccc + +string repeat -n 5 --max 4 123 456 789 +# CHECK: 1231 +# CHECK: 4564 +# CHECK: 7897 + +string repeat -n 5 --max 4 123 '' 789 +# CHECK: 1231 +# CHECK: +# CHECK: 7897 + +# Historical string repeat behavior is no newline if no output. +echo -n before +string repeat -n 5 '' +echo after +# CHECK: beforeafter + string repeat -n-1 foo; and echo "exit 0" # CHECKERR: string repeat: Invalid count value '-1' From c76074b1d6cc556120864437e57edbe6077d8bc9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Cl=C3=A9ment=20Martinez?= Date: Tue, 12 Jan 2021 02:37:38 +0100 Subject: [PATCH 105/105] Add `losetup` completions --- share/completions/losetup.fish | 48 ++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 share/completions/losetup.fish diff --git a/share/completions/losetup.fish b/share/completions/losetup.fish new file mode 100644 index 000000000..0a57c7370 --- /dev/null +++ b/share/completions/losetup.fish @@ -0,0 +1,48 @@ +# losetup - Set up and control loop devices. +# +# This is part of the util-linux package. +# https://www.kernel.org/pub/linux/utils/util-linux + +function __fish_print_losetup_list_output + printf "%s\t%s\n" \ + NAME "Loop device name" \ + AUTOCLEAR "Autoclear flag set" \ + BACK-FILE "Device backing file" \ + BACK-INO "Backing file inode number" \ + BACK-MAJ:MIN "Backing file major:minor device number" \ + MAJ:MIN "Loop device major:minor number" \ + OFFSET "Offset from the beginning" \ + PARTSCAN "Partscan flag set" \ + RO "Read-only device" \ + SIZELIMIT "Size limit of the file in bytes" \ + DIO "Access backing file with direct-io" \ + LOG-SEC "Logical sector size in bytes" +end + +function __fish_print_losetup_attached + losetup --list --raw --noheadings --output NAME,BACK-FILE | string replace ' ' \t +end + +complete -c losetup -s a -l all -d "List all used devices" +complete -c losetup -s d -l detach -x -a "(__fish_print_losetup_attached)" -d "Detach one or more devices" +complete -c losetup -s D -l detach-all -d "Detach all used devices" +complete -c losetup -s f -l find -d "Find first unused device" +complete -c losetup -s c -l set-capacity -x -a "(__fish_print_losetup_attached)" -d "Resize the device" +complete -c losetup -s j -l associated -r -d "List all devices associated with given file" +complete -c losetup -s L -l nooverlap -d "Avoid possible conflict between devices" +complete -c losetup -s o -l offset -x -d "Start at given offset into file" +complete -c losetup -l sizelimit -x -d "Device is limited to give bytes of the file" +complete -c losetup -s b -l sector-size -x -d "Set the logical sector size" +complete -c losetup -s P -l partscan -d "Create a partitioned loop device" +complete -c losetup -s r -l read-only -d "Set up a read-only loop device" +complete -c losetup -l direct-io -x -a "on off" -d "open backing file with O_DIRECT" +complete -c losetup -l show -d "Print device name after setup" +complete -c losetup -s v -l verbose -d "Verbose mode" +complete -c losetup -s J -l json -d "Use JSON --list output format" +complete -c losetup -s l -l list -d "List info about all or specified" +complete -c losetup -s n -l noheadings -d "Don't print headings for --list output" +complete -c losetup -s O -l output -x -a "(__fish_complete_list , __fish_print_losetup_list_output)" -d "Specify columns to output for --list" +complete -c losetup -l output-all -d "Output all columns" +complete -c losetup -l raw -d "Use raw --list output format" +complete -c losetup -s h -l help -d "Display help" +complete -c losetup -s V -l version -d "Display version"