From 904936789b77a44082b8610d7f85d8de84447245 Mon Sep 17 00:00:00 2001 From: Andrew Toskin Date: Wed, 9 Aug 2017 00:46:12 -0700 Subject: [PATCH 01/79] Clarify dependencies: required vs optional, and build vs runtime. A first pass at updating the dependency documentation, based on the discussion in this thread: https://github.com/fish-shell/fish-shell/issues/2062 --- README.md | 43 +++++++++++++++++++++---------------------- 1 file changed, 21 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 24993bc81..f66ac8de4 100644 --- a/README.md +++ b/README.md @@ -13,17 +13,32 @@ Detailed user documentation is available by running `help` within fish, and also ## Building -fish requires a C++11 compiler. It builds successfully with g++ 4.8 or later, or with clang 3.3 or later. +### Dependencies -fish can be built using autotools or Xcode. autoconf 2.60 or later, as well as automake 1.13 or later, are required to build from git versions. These are not required to build from released tarballs. +Compiling fish requires: -fish depends on a curses implementation, such as ncurses. The headers and libraries are required for building. +* a C++11 compiler. It builds successfully with g++ 4.8 or later, or with clang 3.3 or later. fish can be built using autotools or Xcode. autoconf 2.60 or later, as well as automake 1.13 or later, are required to build from git versions, but these are not required to build from the officially released tarballs. +* the headers and libraries of a curses implementation such as ncurses. +* gettext, for translation support. +* Doxygen 1.8.7 or newer, for building the documentation. -fish requires PCRE2 due to the regular expression support contained in the `string` builtin. A copy is included with the source code, and will be used automatically if it does not already exist on your system. +At runtime, fish requires: -fish requires gettext for translation support. +* a curses implementation, such as ncurses. +* gettext, if you need the localizations. +* PCRE2, due to the regular expression support contained in the `string` builtin. A copy is included with the source code, and will be used automatically if it does not already exist on your system. +* a number of common UNIX utilities: coreutils (at least ls, seq, rm, mktemp), hostname, grep, awk, sed, and getopt. +* bc, the "basic calculator" program. -Building the documentation requires Doxygen 1.8.7 or newer. +Optional functionality should degrade gracefully. If fish ever threw an error about missing dependencies for an optional feature, it would be a bug in fish and should be reported on our GitHub. Optional features and dependencies include: + +* fish's builtin `test` syntax, which requires `diff` and `expect`. You don't need these, though, if you don't want to use our tests. +* dynamically generating usage tips for builtin functions, which requires man, apropos, nroff, and ul. +* manual page parsing for auto-completion for external commands, which also requires the above man tools, plus Python 2.6+ or Python 3.3+. Python 2 will also need the backports.lzma package; Python 3.3+ should include the required module by default. +* the web configuration tool, which requires Python 2.6+ or 3.3+. +* the fish_clipboard_* functions (bound to \cv and \cx), which require xsel or pbcopy/pbpaste. + +fish can also show VCS information in the prompt, when the current working directory is in a VCS-tracked project. This is one of the weakest dependencies, though: if you don't use git, you don't need git information in the prompt, and if you do use git, you already have it. ### Autotools Build @@ -55,22 +70,6 @@ On RedHat, CentOS, or Amazon EC2: sudo yum install ncurses-devel -## Runtime Dependencies - -fish requires a curses implementation, such as ncurses, to run. - -fish requires PCRE2 due to the regular expression support contained in the `string` builtin. A bundled version will be compiled in automatically at build time if required. - -fish requires a number of utilities to operate, which should be present on any Unix, GNU/Linux or OS X system. These include (but are not limited to) hostname, grep, awk, sed, and getopt. fish also requires the bc program. - -Translation support requires the gettext program. - -Usage output for builtin functions is generated on-demand from the installed manpages using `nroff` and `ul`. - -Some optional features of fish, such as the manual page completion parser and the web configuration tool, require Python. - -In order to generate completions from man pages compressed with either lzma or xz, you may need to install an extra Python package. Python versions prior to 2.6 are not supported. To process lzma-compresed manpages, backports.lzma is needed for Python 3.2 or older. From version 3.3 onwards, Python already includes the required module. - ## Packages for Linux Instructions on how to find builds for several Linux distros are at From 09cb31a172773761a305f7353b44e9bb056a5d98 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Thu, 10 Aug 2017 00:37:32 +0200 Subject: [PATCH 02/79] Fix clearing abandoned line with VTE (#4243) * Fix clearing abandoned line with VTE With VTE-based terminals, resizing currently causes multi-line prompts to go weird. This changes the sequence we use to clear the line to one suggested by a VTE developer (https://bugzilla.gnome.org/show_bug.cgi?id=763390#c4). It changes nothing in konsole 17.04.3 and urxvt 9.22, but they already work. Note that this does not fix the case where output did not end in a newline, but that doesn't seem to be up to us. Also, it only affects those lines. Fixes #2320. * Use terminfo definition instead of hardcoding Thanks to @ixjlyons. --- src/screen.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/screen.cpp b/src/screen.cpp index 0d0d3cc1d..7a07d21c5 100644 --- a/src/screen.cpp +++ b/src/screen.cpp @@ -1287,7 +1287,7 @@ void s_reset(screen_t *s, screen_reset_mode_t mode) { // line above your prompt. This doesn't make a difference in normal usage, but copying and // pasting your terminal log becomes a pain. This commit clears that line, making it an // actual empty line. - abandon_line_string.append(L"\e[2K"); + abandon_line_string.append(str2wcstring(clr_eol)); const std::string narrow_abandon_line_string = wcs2string(abandon_line_string); write_loop(STDOUT_FILENO, narrow_abandon_line_string.c_str(), From d84a859f4f15a0b87c931d8d81b1a3e45086fa2f Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Thu, 10 Aug 2017 02:24:13 +0200 Subject: [PATCH 03/79] Revert "Fix clearing abandoned line with VTE (#4243)" Unfortunately, this breaks the expect tests. So, until I can figure out how to unbreak them: This reverts commit 09cb31a172773761a305f7353b44e9bb056a5d98. --- src/screen.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/screen.cpp b/src/screen.cpp index 7a07d21c5..0d0d3cc1d 100644 --- a/src/screen.cpp +++ b/src/screen.cpp @@ -1287,7 +1287,7 @@ void s_reset(screen_t *s, screen_reset_mode_t mode) { // line above your prompt. This doesn't make a difference in normal usage, but copying and // pasting your terminal log becomes a pain. This commit clears that line, making it an // actual empty line. - abandon_line_string.append(str2wcstring(clr_eol)); + abandon_line_string.append(L"\e[2K"); const std::string narrow_abandon_line_string = wcs2string(abandon_line_string); write_loop(STDOUT_FILENO, narrow_abandon_line_string.c_str(), From 781a6b118a0fa7bf7e6bc4f03cfc443fa0c77464 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Wed, 9 Aug 2017 19:54:28 -0700 Subject: [PATCH 04/79] overlooked reference to `.` completion script in makefile --- Makefile.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile.in b/Makefile.in index 08aaa35b3..0ecb86acc 100644 --- a/Makefile.in +++ b/Makefile.in @@ -169,7 +169,7 @@ TEST_IN := $(wildcard tests/test*.in) # # Files in ./share/completions/ # -COMPLETIONS_DIR_FILES := $(wildcard share/completions/*.fish) share/completions/..fish +COMPLETIONS_DIR_FILES := $(wildcard share/completions/*.fish) # # Files in ./share/functions/ From 1c9370dbd2a3a0135c9c5fdd6b2a690dcd0c8027 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Wed, 9 Aug 2017 11:11:58 -0700 Subject: [PATCH 05/79] replace `var_entry_t` with `env_var_t` This is a step to storing fish vars as actual vectors rather than flat strings. --- src/builtin_set.cpp | 2 +- src/env.cpp | 120 ++++++++++++++++------------------- src/env.h | 25 ++++---- src/env_universal_common.cpp | 42 ++++++------ src/env_universal_common.h | 4 +- src/fish_tests.cpp | 7 +- src/highlight.cpp | 4 +- src/path.cpp | 8 +-- tests/read.err | 3 + tests/read.in | 28 ++++---- tests/read.out | 3 + 11 files changed, 120 insertions(+), 126 deletions(-) diff --git a/src/builtin_set.cpp b/src/builtin_set.cpp index cad1034ae..810451033 100644 --- a/src/builtin_set.cpp +++ b/src/builtin_set.cpp @@ -296,7 +296,7 @@ static int my_env_set(const wchar_t *cmd, const wchar_t *key, const wcstring_lis // We don't check `val->empty()` because an array var with a single empty string will be // "empty". A truly empty array var is set to the special value `ENV_NULL`. auto val = list_to_array_val(list); - retval = env_set(key, *val == ENV_NULL ? NULL : val->c_str(), scope | ENV_USER); + retval = env_set(key, val->c_str(), scope | ENV_USER); switch (retval) { case ENV_OK: { retval = STATUS_CMD_OK; diff --git a/src/env.cpp b/src/env.cpp index 736a73433..63389d4ed 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -126,8 +126,8 @@ class env_node_t { /// Pointer to next level. std::unique_ptr next; - /// Returns a pointer to the given entry if present, or NULL. - const var_entry_t *find_entry(const wcstring &key); + /// Returns the given entry if present else env_var_t::missing_var. + const env_var_t find_entry(const wcstring &key); }; class variable_entry_t { @@ -174,27 +174,21 @@ struct var_stack_t { env_node_t *next_scope_to_search(env_node_t *node); const env_node_t *next_scope_to_search(const env_node_t *node) const; - // Returns the scope used for unspecified scopes - // An unspecified scope is either the topmost shadowing scope, or the global scope if none - // This implements the default behavior of 'set' + // Returns the scope used for unspecified scopes. An unspecified scope is either the topmost + // shadowing scope, or the global scope if none. This implements the default behavior of `set`. env_node_t *resolve_unspecified_scope(); }; void var_stack_t::push(bool new_scope) { std::unique_ptr node(new env_node_t(new_scope)); - // Copy local-exported variables + // Copy local-exported variables. auto top_node = top.get(); - // Only if we introduce a new shadowing scope - // i.e. not if it's just `begin; end` or "--no-scope-shadowing". - if (new_scope) { - if (!(top_node == this->global_env)) { - for (auto &var : top_node->env) { - if (var.second.exportv) { - // This should copy var - node->env.insert(var); - } - } + // Only if we introduce a new shadowing scope; i.e. not if it's just `begin; end` or + // "--no-scope-shadowing". + if (new_scope && top_node != this->global_env) { + for (auto &var : top_node->env) { + if (var.second.exportv) node->env.insert(var); } } @@ -240,8 +234,8 @@ void var_stack_t::pop() { assert(this->top != NULL); for (const auto &entry_pair : old_top->env) { - const var_entry_t &entry = entry_pair.second; - if (entry.exportv) { + const env_var_t &var = entry_pair.second; + if (var.exportv) { this->mark_changed_exported(); break; } @@ -309,13 +303,10 @@ static bool is_electric(const wcstring &key) { return env_electric.find(key.c_str()) != env_electric.end(); } -const var_entry_t *env_node_t::find_entry(const wcstring &key) { - const var_entry_t *result = NULL; - var_table_t::const_iterator where = env.find(key); - if (where != env.end()) { - result = &where->second; - } - return result; +const env_var_t env_node_t::find_entry(const wcstring &key) { + var_table_t::const_iterator entry = env.find(key); + if (entry != env.end()) return entry->second; + return env_var_t::missing_var(); } /// Return the current umask value. @@ -959,10 +950,8 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { static env_node_t *env_get_node(const wcstring &key) { env_node_t *env = vars_stack().top.get(); while (env != NULL) { - if (env->find_entry(key) != NULL) { - break; - } - + env_var_t var = env->find_entry(key); + if (!var.missing()) break; env = vars_stack().next_scope_to_search(env); } return env; @@ -1059,8 +1048,8 @@ int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t var_mode) if (preexisting_node != NULL) { var_table_t::const_iterator result = preexisting_node->env.find(key); assert(result != preexisting_node->env.end()); - const var_entry_t &entry = result->second; - if (entry.exportv) { + const env_var_t &var = result->second; + if (var.exportv) { preexisting_entry_exportv = true; has_changed_new = true; } @@ -1074,7 +1063,7 @@ int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t var_mode) } else if (preexisting_node != NULL) { node = preexisting_node; if ((var_mode & (ENV_EXPORT | ENV_UNEXPORT)) == 0) { - // use existing entry's exportv + // Use existing entry's exportv status. var_mode = //!OCLINT(parameter reassignment) preexisting_entry_exportv ? ENV_EXPORT : 0; } @@ -1108,20 +1097,21 @@ int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t var_mode) if (!done) { // Set the entry in the node. Note that operator[] accesses the existing entry, or // creates a new one. - var_entry_t &entry = node->env[key]; - if (entry.exportv) { + env_var_t &var = node->env[key]; + if (var.exportv) { // This variable already existed, and was exported. has_changed_new = true; } - entry.val = val; + + var.set_val(val); if (var_mode & ENV_EXPORT) { // The new variable is exported. - entry.exportv = true; + var.exportv = true; node->exportv = true; has_changed_new = true; } else { - entry.exportv = false; - // Set the node's exportv when it changes something about exports + var.exportv = false; + // Set the node's exported when it changes something about exports // (also when it redefines a variable to not be exported). node->exportv = has_changed_old != has_changed_new; } @@ -1219,7 +1209,7 @@ int env_remove(const wcstring &key, int var_mode) { } wcstring env_var_t::as_string(void) const { - assert(!is_missing); + //assert(!is_missing); return val; } @@ -1276,12 +1266,15 @@ env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { env_node_t *env = search_local ? vars_stack().top.get() : vars_stack().global_env; while (env != NULL) { - const var_entry_t *entry = env->find_entry(key); - if (entry != NULL && (entry->exportv ? search_exported : search_unexported)) { - if (entry->val == ENV_NULL) { - return env_var_t::missing_var(); - } - return entry->val; + const env_var_t var = env->find_entry(key); + if (!var.missing() && (var.exportv ? search_exported : search_unexported)) { + // The following statement is wrong because ENV_NULL is supposed to mean the var is + // set but has zero elements. Therefore we should not return missing_var. However, + // If this is changed to return `var` without the special-case then unit tests fail. + // Note that tokenize_variable_array() explicitly handles a var whose string + // representation contains only ENV_NULL. + if (var.as_string() == ENV_NULL) return env_var_t::missing_var(); + return var; } if (has_scope) { @@ -1342,8 +1335,8 @@ bool env_exist(const wchar_t *key, env_mode_flags_t mode) { var_table_t::const_iterator result = env->env.find(key); if (result != env->env.end()) { - const var_entry_t &res = result->second; - return res.exportv ? test_exported : test_unexported; + const env_var_t var = result->second; + return var.exportv ? test_exported : test_unexported; } env = vars_stack().next_scope_to_search(env); } @@ -1385,9 +1378,9 @@ static void add_key_to_string_set(const var_table_t &envs, std::set *s bool show_exported, bool show_unexported) { var_table_t::const_iterator iter; for (iter = envs.begin(); iter != envs.end(); ++iter) { - const var_entry_t &e = iter->second; + const env_var_t &var = iter->second; - if ((e.exportv && show_exported) || (!e.exportv && show_unexported)) { + if ((var.exportv && show_exported) || (!var.exportv && show_unexported)) { // Insert this key. str_set->insert(iter->first); } @@ -1440,39 +1433,39 @@ wcstring_list_t env_get_names(int flags) { } /// Get list of all exported variables. -static void get_exported(const env_node_t *n, std::map *h) { +static void get_exported(const env_node_t *n, var_table_t &h) { if (!n) return; - if (n->new_scope) + if (n->new_scope) { get_exported(vars_stack().global_env, h); - else + } else { get_exported(n->next.get(), h); + } var_table_t::const_iterator iter; for (iter = n->env.begin(); iter != n->env.end(); ++iter) { const wcstring &key = iter->first; - const var_entry_t &val_entry = iter->second; + const env_var_t var = iter->second; - if (val_entry.exportv && val_entry.val != ENV_NULL) { + if (!var.missing() && var.exportv) { // Export the variable. Don't use std::map::insert here, since we need to overwrite // existing values from previous scopes. - (*h)[key] = val_entry.val; + h[key] = var; } else { // We need to erase from the map if we are not exporting, since a lower scope may have // exported. See #2132. - h->erase(key); + h.erase(key); } } } // Given a map from key to value, add values to out of the form key=value. -static void export_func(const std::map &envs, std::vector &out) { +static void export_func(const var_table_t &envs, std::vector &out) { out.reserve(out.size() + envs.size()); - std::map::const_iterator iter; - for (iter = envs.begin(); iter != envs.end(); ++iter) { + for (auto iter = envs.begin(); iter != envs.end(); ++iter) { const wcstring &key = iter->first; const std::string &ks = wcs2string(key); - std::string vs = wcs2string(iter->second); + std::string vs = wcs2string(iter->second.as_string()); // Arrays in the value are ASCII record separator (0x1e) delimited. But some variables // should have colons. Add those. @@ -1497,11 +1490,10 @@ void var_stack_t::update_export_array_if_necessary() { if (!this->has_changed_exported) { return; } - std::map vals; debug(4, L"env_export_arr() recalc"); - - get_exported(this->top.get(), &vals); + var_table_t vals; + get_exported(this->top.get(), vals); if (uvars()) { const wcstring_list_t uni = uvars()->get_names(true, false); @@ -1544,7 +1536,7 @@ void env_set_argv(const wchar_t *const *argv) { env_set(L"argv", sb.c_str(), ENV_LOCAL); } else { - env_set(L"argv", 0, ENV_LOCAL); + env_set(L"argv", NULL, ENV_LOCAL); } } diff --git a/src/env.h b/src/env.h index b5c8131c5..aee366f6c 100644 --- a/src/env.h +++ b/src/env.h @@ -75,16 +75,19 @@ class env_var_t { bool is_missing; public: + bool exportv; // whether the variable should be exported + static env_var_t missing_var() { env_var_t result((wcstring())); result.is_missing = true; + result.exportv = false; return result; } - env_var_t(const env_var_t &x) : val(x.val), is_missing(x.is_missing) {} - env_var_t(const wcstring &x) : val(x), is_missing(false) {} - env_var_t(const wchar_t *x) : val(x), is_missing(false) {} - env_var_t() : val(L""), is_missing(false) {} + env_var_t(const env_var_t &x) : val(x.val), is_missing(x.is_missing), exportv(x.exportv) {} + env_var_t(const wcstring &x) : val(x), is_missing(false), exportv(false) {} + env_var_t(const wchar_t *x) : val(x), is_missing(false), exportv(false) {} + env_var_t() : val(L""), is_missing(false), exportv(false) {} bool empty(void) const { return val.empty(); }; bool missing(void) const { return is_missing; } @@ -96,10 +99,14 @@ class env_var_t { env_var_t &operator=(const env_var_t &v) { is_missing = v.is_missing; + exportv = v.exportv; val = v.val; return *this; } + void set_val(const wcstring &s) { val = s; is_missing = false; } + void set_val(const wchar_t *s) { val = s; is_missing = false; } + bool operator==(const env_var_t &s) const { return is_missing == s.is_missing && val == s.val; } bool operator==(const wcstring &s) const { return !is_missing && val == s; } @@ -192,15 +199,7 @@ class env_vars_snapshot_t { extern int g_fork_count; extern bool g_use_posix_spawn; -/// A variable entry. Stores the value of a variable and whether it should be exported. -struct var_entry_t { - wcstring val; // the value of the variable - bool exportv; // whether the variable should be exported - - var_entry_t() : exportv(false) {} -}; - -typedef std::map var_table_t; +typedef std::map var_table_t; extern bool term_has_xn; // does the terminal have the "eat_newline_glitch" diff --git a/src/env_universal_common.cpp b/src/env_universal_common.cpp index 00228113f..19a087ee1 100644 --- a/src/env_universal_common.cpp +++ b/src/env_universal_common.cpp @@ -273,7 +273,7 @@ env_var_t env_universal_t::get(const wcstring &name) const { env_var_t result = env_var_t::missing_var(); var_table_t::const_iterator where = vars.find(name); if (where != vars.end()) { - result = env_var_t(where->second.val); + result = where->second; } return result; } @@ -295,10 +295,10 @@ void env_universal_t::set_internal(const wcstring &key, const wcstring &val, boo return; } - var_entry_t *entry = &vars[key]; - if (entry->exportv != exportv || entry->val != val) { - entry->val = val; - entry->exportv = exportv; + env_var_t &entry = vars[key]; + if (entry.exportv != exportv || entry.as_string() != val) { + entry.set_val(val); + entry.exportv = exportv; // If we are overwriting, then this is now modified. if (overwrite) { @@ -332,8 +332,8 @@ wcstring_list_t env_universal_t::get_names(bool show_exported, bool show_unexpor var_table_t::const_iterator iter; for (iter = vars.begin(); iter != vars.end(); ++iter) { const wcstring &key = iter->first; - const var_entry_t &e = iter->second; - if ((e.exportv && show_exported) || (!e.exportv && show_unexported)) { + const env_var_t &var = iter->second; + if ((var.exportv && show_exported) || (!var.exportv && show_unexported)) { result.push_back(key); } } @@ -369,18 +369,18 @@ void env_universal_t::generate_callbacks(const var_table_t &new_vars, } // See if the value has changed. - const var_entry_t &new_entry = iter->second; + const env_var_t &new_entry = iter->second; var_table_t::const_iterator existing = this->vars.find(key); if (existing == this->vars.end() || existing->second.exportv != new_entry.exportv || - existing->second.val != new_entry.val) { + existing->second != new_entry) { // Value has changed. callbacks.push_back( - callback_data_t(new_entry.exportv ? SET_EXPORT : SET, key, new_entry.val)); + callback_data_t(new_entry.exportv ? SET_EXPORT : SET, key, new_entry.as_string())); } } } -void env_universal_t::acquire_variables(var_table_t *vars_to_acquire) { +void env_universal_t::acquire_variables(var_table_t &vars_to_acquire) { // Copy modified values from existing vars to vars_to_acquire. for (std::set::iterator iter = this->modified.begin(); iter != this->modified.end(); ++iter) { @@ -388,19 +388,19 @@ void env_universal_t::acquire_variables(var_table_t *vars_to_acquire) { var_table_t::iterator src_iter = this->vars.find(key); if (src_iter == this->vars.end()) { /* The value has been deleted. */ - vars_to_acquire->erase(key); + vars_to_acquire.erase(key); } else { // The value has been modified. Copy it over. Note we can destructively modify the // source entry in vars since we are about to get rid of this->vars entirely. - var_entry_t &src = src_iter->second; - var_entry_t &dst = (*vars_to_acquire)[key]; - dst.val = std::move(src.val); + env_var_t &src = src_iter->second; + env_var_t &dst = vars_to_acquire[key]; + dst.set_val(src.as_string()); dst.exportv = src.exportv; } } // We have constructed all the callbacks and updated vars_to_acquire. Acquire it! - this->vars = std::move(*vars_to_acquire); + this->vars = std::move(vars_to_acquire); } void env_universal_t::load_from_fd(int fd, callback_data_list_t &callbacks) { @@ -418,7 +418,7 @@ void env_universal_t::load_from_fd(int fd, callback_data_list_t &callbacks) { this->generate_callbacks(new_vars, callbacks); // Acquire the new variables. - this->acquire_variables(&new_vars); + this->acquire_variables(new_vars); last_read_file = current_file; } } @@ -464,8 +464,8 @@ bool env_universal_t::write_to_fd(int fd, const wcstring &path) { // Append the entry. Note that append_file_entry may fail, but that only affects one // variable; soldier on. const wcstring &key = iter->first; - const var_entry_t &entry = iter->second; - append_file_entry(entry.exportv ? SET_EXPORT : SET, key, entry.val, &contents, &storage); + const env_var_t &var = iter->second; + append_file_entry(var.exportv ? SET_EXPORT : SET, key, var.as_string(), &contents, &storage); // Go to next. ++iter; @@ -845,9 +845,9 @@ void env_universal_t::parse_message_internal(const wcstring &msgstr, var_table_t wcstring val; if (unescape_string(tmp + 1, &val, 0)) { - var_entry_t &entry = (*vars)[key]; + env_var_t &entry = (*vars)[key]; entry.exportv = exportv; - entry.val = std::move(val); // acquire the value + entry.set_val(val); // acquire the value } } else { debug(1, PARSE_ERR, msg); diff --git a/src/env_universal_common.h b/src/env_universal_common.h index a73a57c43..bca672ac7 100644 --- a/src/env_universal_common.h +++ b/src/env_universal_common.h @@ -60,9 +60,9 @@ class env_universal_t { // the new vars. void generate_callbacks(const var_table_t &new_vars, callback_data_list_t &callbacks) const; - // Given a variable table, copy unmodified values into self. May destructively modified + // Given a variable table, copy unmodified values into self. May destructively modify // vars_to_acquire. - void acquire_variables(var_table_t *vars_to_acquire); + void acquire_variables(var_table_t &vars_to_acquire); static void parse_message_internal(const wcstring &msg, var_table_t *vars, wcstring *storage); static var_table_t read_message_internal(int fd); diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 09e57e091..9c539d96e 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -215,6 +215,7 @@ static void popd() { /// Test that the fish functions for converting strings to numbers work. static void test_str_to_num() { + say(L"Testing str_to_num"); const wchar_t *end; int i; long l; @@ -4368,6 +4369,9 @@ int main(int argc, char **argv) { // Set default signal handlers, so we can ctrl-C out of this. signal_reset_handlers(); + if (should_test_function("utility_functions")) test_utility_functions(); + if (should_test_function("wcstring_tok")) test_wcstring_tok(); + if (should_test_function("env_vars")) test_env_vars(); if (should_test_function("str_to_num")) test_str_to_num(); if (should_test_function("highlighting")) test_highlighting(); if (should_test_function("new_parser_ll2")) test_new_parser_ll2(); @@ -4409,15 +4413,12 @@ int main(int argc, char **argv) { if (should_test_function("autosuggestion_ignores")) test_autosuggestion_ignores(); if (should_test_function("autosuggestion_combining")) test_autosuggestion_combining(); if (should_test_function("autosuggest_suggest_special")) test_autosuggest_suggest_special(); - if (should_test_function("wcstring_tok")) test_wcstring_tok(); if (should_test_function("history")) history_tests_t::test_history(); if (should_test_function("history_merge")) history_tests_t::test_history_merge(); if (should_test_function("history_races")) history_tests_t::test_history_races(); if (should_test_function("history_formats")) history_tests_t::test_history_formats(); if (should_test_function("string")) test_string(); - if (should_test_function("env_vars")) test_env_vars(); if (should_test_function("illegal_command_exit_code")) test_illegal_command_exit_code(); - if (should_test_function("utility_functions")) test_utility_functions(); // history_tests_t::test_history_speed(); say(L"Encountered %d errors in low-level tests", err_count); diff --git a/src/highlight.cpp b/src/highlight.cpp index 2a64336f9..aefafa436 100644 --- a/src/highlight.cpp +++ b/src/highlight.cpp @@ -279,7 +279,6 @@ rgb_color_t highlight_get_color(highlight_spec_t highlight, bool is_background) // Handle modifiers. if (highlight & highlight_modifier_valid_path) { env_var_t var2 = env_get(L"fish_color_valid_path"); - const wcstring val2 = var2.missing() ? L"" : var2.c_str(); rgb_color_t result2 = parse_color(var2, is_background); if (result.is_normal()) @@ -1014,8 +1013,9 @@ static bool command_is_valid(const wcstring &cmd, enum parse_statement_decoratio if (!is_valid && command_ok) is_valid = path_get_path(cmd, NULL, vars); // Implicit cd - if (!is_valid && implicit_cd_ok) + if (!is_valid && implicit_cd_ok) { is_valid = path_can_be_implicit_cd(cmd, NULL, working_directory.c_str(), vars); + } // Return what we got. return is_valid; diff --git a/src/path.cpp b/src/path.cpp index bee6ed8b1..e2c7216af 100644 --- a/src/path.cpp +++ b/src/path.cpp @@ -48,20 +48,18 @@ static bool path_get_path_core(const wcstring &cmd, wcstring *out_path, } int err = ENOENT; - wcstring bin_path; + wcstring_list_t pathsv; if (!bin_path_var.missing()) { - bin_path = bin_path_var.as_string(); + bin_path_var.to_list(pathsv); } else { // Note that PREFIX is defined in the `Makefile` and is thus defined when this module is // compiled. This ensures we always default to "/bin", "/usr/bin" and the bin dir defined // for the fish programs. Possibly with a duplicate dir if PREFIX is empty, "/", "/usr" or // "/usr/". If the PREFIX duplicates /bin or /usr/bin that is harmless other than a trivial // amount of time testing a path we've already tested. - bin_path = *list_to_array_val(wcstring_list_t({L"/bin", L"/usr/bin", PREFIX L"/bin"})); + pathsv = wcstring_list_t({L"/bin", L"/usr/bin", PREFIX L"/bin"}); } - wcstring_list_t pathsv; - bin_path_var.to_list(pathsv); for (auto next_path : pathsv) { if (next_path.empty()) continue; append_path_component(next_path, cmd); diff --git a/tests/read.err b/tests/read.err index 676c6e2d1..d76028bc0 100644 --- a/tests/read.err +++ b/tests/read.err @@ -8,6 +8,9 @@ read: Expected at least 1 args, got only 0 read: Expected 1 args, got 0 read: Expected 1 args, got 2 +#################### +# Verify correct behavior of subcommands and splitting of input. + #################### # Test splitting input diff --git a/tests/read.in b/tests/read.in index 36cbcc98e..cd5bfa07e 100644 --- a/tests/read.in +++ b/tests/read.in @@ -3,8 +3,6 @@ # Test read builtin and IFS. # -########### -# Start by testing that invocation errors are handled correctly. logmsg Read with no vars is an error read @@ -13,19 +11,19 @@ read -a read -a v1 v2 read -a v1 -########### -# Verify correct behavior of subcommands and splitting of input. -count (echo one\ntwo) -set -l IFS \t -count (echo one\ntwo) -set -l IFS -count (echo one\ntwo) -echo [(echo -n one\ntwo)] -count (echo one\ntwo\n) -echo [(echo -n one\ntwo\n)] -count (echo one\ntwo\n\n) -echo [(echo -n one\ntwo\n\n)] -set -le IFS +logmsg Verify correct behavior of subcommands and splitting of input. +begin + count (echo one\ntwo) + set -l IFS \t + count (echo one\ntwo) + set -l IFS + count (echo one\ntwo) + echo [(echo -n one\ntwo)] + count (echo one\ntwo\n) + echo [(echo -n one\ntwo\n)] + count (echo one\ntwo\n\n) + echo [(echo -n one\ntwo\n\n)] +end function print_vars --no-scope-shadowing set -l space diff --git a/tests/read.out b/tests/read.out index 4b495bdc3..eb1c23a6d 100644 --- a/tests/read.out +++ b/tests/read.out @@ -4,6 +4,9 @@ #################### # Read with -a and anything other than exactly on var name is an error + +#################### +# Verify correct behavior of subcommands and splitting of input. 2 2 1 From 9b6256d0fc62883efa31a0abd0cfee6f83eb4f51 Mon Sep 17 00:00:00 2001 From: David Adam Date: Thu, 10 Aug 2017 13:14:20 +0800 Subject: [PATCH 06/79] docs: improve set -Ux language and example By far the most common problem with universal variables being overridden by global variables is other values being imported from the environment; the `set -q; or set -gx` is much more of an edge case. --- doc_src/faq.hdr | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/doc_src/faq.hdr b/doc_src/faq.hdr index 320b4e91e..4b319e202 100644 --- a/doc_src/faq.hdr +++ b/doc_src/faq.hdr @@ -14,7 +14,7 @@ - How do I run a subcommand? The backtick doesn't work! - How do I get the exit status of a command? - How do I set an environment variable for just one command? -- Why doesn't `set -Ux` (exported universal vars) seem to work? +- Why doesn't `set -Ux` (exported universal varables) seem to work? - How do I customize my syntax highlighting colors? - How do I update man page completions? - Why does cd, pwd and other fish commands always resolve symlinked directories to their canonical path? @@ -126,19 +126,27 @@ begin end \endfish -\section faq-exported-uvar Why doesn't `set -Ux` (exported universal vars) seem to work? +\section faq-exported-uvar Why doesn't `set -Ux` (exported universal variables) seem to work? -Lots of users try to set exported environment variables like `EDITOR` and `TZ` as universal variables; e.g., `set -Ux`. That works but the behavior can be surprising. Keep in mind that when resolving a variable reference (e.g., `echo $EDITOR`) fish first looks in local scope, then global scope, and finally universal scope. Also keep in mind that environment vars imported when fish starts running are placed in the global scope. So if `EDITOR` or `TZ` is already in the environment when fish starts running your universal var by the same name is not used. +A global variable of the same name already exists. -The recommended practice is to not export universal variables in the hope they will be present in all future shells. Instead place statements like the following example in your config.fish file: +Environment variables such as `EDITOR` or `TZ` can be set universally using `set -Ux`. However, if +there is an environment variable already set before fish starts (such as by login scripts or system +administrators), it is imported into fish as a global variable. The variable scopes are searched from the "inside out", which +means that local variables are checked first, followed by global variables, and finally universal +variables. + +This means that the global value takes precedence over the universal value. + +To avoid this problem, consider changing the setting which fish inherits. If this is not possible, +add a statement to your user initilization file (usually +`~/.config/fish/config.fish`): \fish{cli-dark} -set -q EDITOR -or set -gx EDITOR vim +set -gx EDITOR vim \endfish -Now when fish starts it will use the existing value for the variable if it was in the environment. Otherwise it will be set to your preferred default. This allows programs like emacs or an IDE to start a fish shell with a different value for the var. This is effectively the same behavior seen when setting it as a uvar but is explicit and therefore easier to reason about and debug. If you don't want to allow using the existing environment value just unconditionally `set -gx` the var in your config.fish file. - \section faq-customize-colors How do I customize my syntax highlighting colors? Use the web configuration tool, `fish_config`, or alter the `fish_color` family of environment variables. From 559b05d01df07f2d60caaa686d4dd1357ab231bb Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Wed, 9 Aug 2017 11:11:58 -0700 Subject: [PATCH 07/79] fix bug in `env_get()` involving empty vars My previous change to eliminate `class var_entry_t` caused me to notice that `env_get()` turned a set but empty var into a missing var. Which is wrong. Fixing that brought to light several other pieces of code that were wrong as a consequence of the aforementioned bug. Another step to fixing issue #4200. --- Makefile.in | 4 ++-- src/builtin_argparse.cpp | 7 +++---- src/builtin_read.cpp | 2 +- src/builtin_set.cpp | 8 +++----- src/common.cpp | 1 - src/env.cpp | 6 ------ src/env.h | 4 ++-- src/expand.cpp | 20 +++++++++----------- src/expand.h | 2 +- src/fish.cpp | 2 +- 10 files changed, 22 insertions(+), 34 deletions(-) diff --git a/Makefile.in b/Makefile.in index 0ecb86acc..513eaf33f 100644 --- a/Makefile.in +++ b/Makefile.in @@ -353,7 +353,7 @@ test: test-prep install-force test_low_level test_high_level # We want the various tests to run serially so their output doesn't mix # We can do that by adding ordering dependencies based on what goals are being used. # -test_goals := test_low_level test_invocation test_fishscript test_interactive +test_goals := test_low_level test_fishscript test_interactive test_invocation # # The following variables define targets that depend on the tests. If any more targets @@ -374,7 +374,7 @@ test_low_level: fish_tests $(call filter_up_to,test_low_level,$(active_test_goal test_high_level: DESTDIR = $(PWD)/test/root/ test_high_level: prefix = . -test_high_level: test-prep install-force test_invocation test_fishscript test_interactive +test_high_level: test-prep install-force test_fishscript test_interactive test_invocation .PHONY: test_high_level test_invocation: $(call filter_up_to,test_invocation,$(active_test_goals)) diff --git a/src/builtin_argparse.cpp b/src/builtin_argparse.cpp index 45475219d..55e7afc4d 100644 --- a/src/builtin_argparse.cpp +++ b/src/builtin_argparse.cpp @@ -645,8 +645,7 @@ static void set_argparse_result_vars(argparse_cmd_opts_t &opts) { auto val = list_to_array_val(opt_spec->vals); if (opt_spec->short_flag_valid) { - env_set(var_name_prefix + opt_spec->short_flag, *val == ENV_NULL ? NULL : val->c_str(), - ENV_LOCAL); + env_set(var_name_prefix + opt_spec->short_flag, val->c_str(), ENV_LOCAL); } if (!opt_spec->long_flag.empty()) { // We do a simple replacement of all non alphanum chars rather than calling @@ -655,12 +654,12 @@ static void set_argparse_result_vars(argparse_cmd_opts_t &opts) { for (size_t pos = 0; pos < long_flag.size(); pos++) { if (!iswalnum(long_flag[pos])) long_flag[pos] = L'_'; } - env_set(var_name_prefix + long_flag, *val == ENV_NULL ? NULL : val->c_str(), ENV_LOCAL); + env_set(var_name_prefix + long_flag, val->c_str(), ENV_LOCAL); } } auto val = list_to_array_val(opts.argv); - env_set(L"argv", *val == ENV_NULL ? NULL : val->c_str(), ENV_LOCAL); + env_set(L"argv", val->c_str(), ENV_LOCAL); } /// The argparse builtin. This is explicitly not compatible with the BSD or GNU version of this diff --git a/src/builtin_read.cpp b/src/builtin_read.cpp index cfd32ace2..7d6a0522a 100644 --- a/src/builtin_read.cpp +++ b/src/builtin_read.cpp @@ -479,7 +479,7 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { split_about(buff.begin(), buff.end(), opts.delimiter.begin(), opts.delimiter.end(), &splits, LONG_MAX); auto val = list_to_array_val(splits); - env_set(argv[0], *val == ENV_NULL ? NULL : val->c_str(), opts.place); + env_set(argv[0], val->c_str(), opts.place); } } else { // not array if (!opts.have_delimiter) { diff --git a/src/builtin_set.cpp b/src/builtin_set.cpp index 810451033..1d1af95a6 100644 --- a/src/builtin_set.cpp +++ b/src/builtin_set.cpp @@ -457,17 +457,15 @@ static int builtin_set_list(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, if (!names_only) { env_var_t var = env_get(key, compute_scope(opts)); - if (!var.missing()) { + if (!var.missing_or_empty()) { bool shorten = false; - wcstring val = var.as_string(); + wcstring val = expand_escape_variable(var); if (opts.shorten_ok && val.length() > 64) { shorten = true; val.resize(60); } - - wcstring e_value = expand_escape_variable(val); streams.out.append(L" "); - streams.out.append(e_value); + streams.out.append(val); if (shorten) streams.out.push_back(ellipsis_char); } diff --git a/src/common.cpp b/src/common.cpp index e568043f6..68cd2274d 100644 --- a/src/common.cpp +++ b/src/common.cpp @@ -957,7 +957,6 @@ static void escape_string_script(const wchar_t *orig_in, size_t in_len, wcstring case L'\\': case L'\'': { need_escape = need_complex_escape = 1; - // WTF if (escape_all) out += L'\\'; out += L'\\'; out += *in; break; diff --git a/src/env.cpp b/src/env.cpp index 63389d4ed..1fa680d6b 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -1268,12 +1268,6 @@ env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { while (env != NULL) { const env_var_t var = env->find_entry(key); if (!var.missing() && (var.exportv ? search_exported : search_unexported)) { - // The following statement is wrong because ENV_NULL is supposed to mean the var is - // set but has zero elements. Therefore we should not return missing_var. However, - // If this is changed to return `var` without the special-case then unit tests fail. - // Note that tokenize_variable_array() explicitly handles a var whose string - // representation contains only ENV_NULL. - if (var.as_string() == ENV_NULL) return env_var_t::missing_var(); return var; } diff --git a/src/env.h b/src/env.h index aee366f6c..5c5890991 100644 --- a/src/env.h +++ b/src/env.h @@ -89,9 +89,9 @@ class env_var_t { env_var_t(const wchar_t *x) : val(x), is_missing(false), exportv(false) {} env_var_t() : val(L""), is_missing(false), exportv(false) {} - bool empty(void) const { return val.empty(); }; + bool empty(void) const { return val.empty() || val == ENV_NULL; }; bool missing(void) const { return is_missing; } - bool missing_or_empty(void) const { return missing() || val.empty(); } + bool missing_or_empty(void) const { return missing() || empty(); } const wchar_t *c_str(void) const; void to_list(wcstring_list_t &out) const; diff --git a/src/expand.cpp b/src/expand.cpp index f265485d4..4a34ba4a2 100644 --- a/src/expand.cpp +++ b/src/expand.cpp @@ -27,8 +27,10 @@ #include #include +#include #include // IWYU pragma: keep #include +#include #include #include "common.h" @@ -42,7 +44,6 @@ #include "parse_util.h" #include "path.h" #include "proc.h" -#include "util.h" #include "wildcard.h" #include "wutil.h" // IWYU pragma: keep #ifdef KERN_PROCARGS2 @@ -169,15 +170,13 @@ static int is_quotable(const wchar_t *str) { static int is_quotable(const wcstring &str) { return is_quotable(str.c_str()); } wcstring expand_escape_variable(const env_var_t &var) { - wcstring_list_t lst; wcstring buff; + wcstring_list_t lst; var.to_list(lst); - - size_t size = lst.size(); - if (size == 0) { - buff.append(L"''"); - } else if (size == 1) { + if (lst.size() == 0) { + ; // empty list expands to nothing + } else if (lst.size() == 1) { const wcstring &el = lst.at(0); if (el.find(L' ') != wcstring::npos && is_quotable(el)) { @@ -201,6 +200,7 @@ wcstring expand_escape_variable(const env_var_t &var) { } } } + return buff; } @@ -265,9 +265,7 @@ wcstring process_iterator_t::name_for_pid(pid_t pid) { } args = (char *)malloc(maxarg); - if (args == NULL) { // cppcheck-suppress memleak - return result; - } + if (!args) return result; mib[0] = CTL_KERN; mib[1] = KERN_PROCARGS2; @@ -1584,7 +1582,7 @@ bool fish_xdm_login_hack_hack_hack_hack(std::vector *cmds, int argc } std::map abbreviations; -void update_abbr_cache(const wchar_t *op, const wcstring varname) { +void update_abbr_cache(const wchar_t *op, const wcstring &varname) { wcstring abbr; if (!unescape_string(varname.substr(wcslen(L"_fish_abbr_")), &abbr, 0, STRING_STYLE_VAR)) { debug(1, L"Abbreviation var '%ls' is not correctly encoded, ignoring it.", varname.c_str()); diff --git a/src/expand.h b/src/expand.h index 6643b8742..20181a59f 100644 --- a/src/expand.h +++ b/src/expand.h @@ -134,7 +134,7 @@ wcstring replace_home_directory_with_tilde(const wcstring &str); /// Abbreviation support. Expand src as an abbreviation, returning true if one was found, false if /// not. If result is not-null, returns the abbreviation by reference. -void update_abbr_cache(const wchar_t *op, const wcstring varnam); +void update_abbr_cache(const wchar_t *op, const wcstring &varname); bool expand_abbreviation(const wcstring &src, wcstring *output); // Terrible hacks diff --git a/src/fish.cpp b/src/fish.cpp index 096805a6e..a3f45dfe3 100644 --- a/src/fish.cpp +++ b/src/fish.cpp @@ -421,7 +421,7 @@ int main(int argc, char **argv) { list.push_back(str2wcstring(*ptr)); } auto val = list_to_array_val(list); - env_set(L"argv", *val == ENV_NULL ? NULL : val->c_str(), 0); + env_set(L"argv", val->c_str(), ENV_DEFAULT); const wcstring rel_filename = str2wcstring(file); From 591449aba72f5f6a8fa646f57704ece4428eccb4 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Thu, 10 Aug 2017 15:20:53 -0700 Subject: [PATCH 08/79] remove more ENV_NULL references --- src/builtin_set.cpp | 2 -- src/expand.cpp | 10 +--------- 2 files changed, 1 insertion(+), 11 deletions(-) diff --git a/src/builtin_set.cpp b/src/builtin_set.cpp index 1d1af95a6..8672e6fcc 100644 --- a/src/builtin_set.cpp +++ b/src/builtin_set.cpp @@ -293,8 +293,6 @@ static int my_env_set(const wchar_t *cmd, const wchar_t *key, const wcstring_lis if (retval != STATUS_CMD_OK) return retval; } - // We don't check `val->empty()` because an array var with a single empty string will be - // "empty". A truly empty array var is set to the special value `ENV_NULL`. auto val = list_to_array_val(list); retval = env_set(key, val->c_str(), scope | ENV_USER); switch (retval) { diff --git a/src/expand.cpp b/src/expand.cpp index 4a34ba4a2..2bcc6ec18 100644 --- a/src/expand.cpp +++ b/src/expand.cpp @@ -1172,16 +1172,8 @@ static void expand_home_directory(wcstring &input) { if (username.empty()) { // Current users home directory. home = env_get(L"HOME"); - // If home is either missing or empty, - // treat it like an empty list. - // $HOME is defined to be a _path_, - // and those cannot be empty. - // - // We do not expand a string-empty var differently, - // because that results in bogus paths - // - ~/foo turns into /foo. if (home.missing_or_empty()) { - input = ENV_NULL; + input.clear(); return; } tail_idx = 1; From 7f1bdc5541ae721504748a3b431f9985a8bf0faa Mon Sep 17 00:00:00 2001 From: peoro Date: Wed, 9 Aug 2017 11:56:04 +0200 Subject: [PATCH 09/79] Improved warning message when exiting with jobs still active Fixes #4303 --- src/reader.cpp | 18 ++++++++++++++++-- tests/exit.expect | 19 +++++++++++++++---- 2 files changed, 31 insertions(+), 6 deletions(-) diff --git a/src/reader.cpp b/src/reader.cpp index c46d13af8..1855fece9 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -2190,6 +2190,21 @@ bool shell_is_exiting() { return end_loop; } +static void bg_job_warning() { + fputws(_(L"There are still jobs active:\n"), stdout); + fputws(_(L"\n PID Command\n"), stdout); + + job_iterator_t jobs; + while (job_t *j = jobs.next()) { + if (!job_is_completed(j)) { + fwprintf(stdout, L"%6d %ls\n", j->pgid, j->command_wcstr()); + } + } + fputws(L"\n", stdout); + fputws(_(L"Use `disown PID` to let them live independently from fish.\n"), stdout); + fputws(_(L"A second attempt to exit will terminate them.\n"), stdout); +} + /// This function is called when the main loop notices that end_loop has been set while in /// interactive mode. It checks if it is ok to exit. static void handle_end_loop() { @@ -2214,8 +2229,7 @@ static void handle_end_loop() { } if (!data->prev_end_loop && bg_jobs) { - fputws(_(L"There are still jobs active (use the jobs command to see them).\n"), stdout); - fputws(_(L"A second attempt to exit will terminate them.\n"), stdout); + bg_job_warning(); reader_exit(0, 0); data->prev_end_loop = 1; return; diff --git a/tests/exit.expect b/tests/exit.expect index 3ff2074f7..6c0756675 100644 --- a/tests/exit.expect +++ b/tests/exit.expect @@ -9,8 +9,13 @@ expect_prompt send "sleep 111 &\r" expect_prompt send "exit\r" -expect "There are still jobs active" -expect "A second attempt to exit will terminate them." +expect -re "There are still jobs active:\r +\r + PID Command\r + *\\d+ sleep 111 &\r +\r +Use `disown PID` to let them live independently from fish.\r +A second attempt to exit will terminate them.\r" expect_prompt # Running anything other than `exit` should result in the same warning with @@ -18,8 +23,14 @@ expect_prompt send "sleep 113 &\r" expect_prompt send "exit\r" -expect "There are still jobs active" -expect "A second attempt to exit will terminate them." +expect -re "There are still jobs active:\r +\r + PID Command\r + *\\d+ sleep 113 &\r + *\\d+ sleep 111 &\r +\r +Use `disown PID` to let them live independently from fish.\r +A second attempt to exit will terminate them.\r" expect_prompt # Verify that asking to exit a second time does so. From 101926a8e8e3a131d50128362be6fb303638e06f Mon Sep 17 00:00:00 2001 From: Andrew Toskin Date: Thu, 10 Aug 2017 23:03:51 -0700 Subject: [PATCH 10/79] Clarify notes on dependency errors, tests, and VCS integration. An optional feature that suggests you install Python is okay; core-dumping is not. The note on tests was about fish development tests, not the `test` builtin for conditional syntax. Specifically mention git, hg, and svn in the VCS section. --- CONTRIBUTING.md | 2 ++ README.md | 5 ++--- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 373c3dc9a..a01431df2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -31,6 +31,8 @@ To make linting the code easy there are two make targets: `lint` and `lint-all`. Fish has custom cppcheck rules in the file `.cppcheck.rule`. These help catch mistakes such as using `wcwidth()` rather than `fish_wcwidth()`. Please add a new rule if you find similar mistakes being made. +Fish also depends on `diff` and `expect` for its tests. + ### Dealing With Lint Warnings You are strongly encouraged to address a lint warning by refactoring the code, changing variable names, or whatever action is implied by the warning. diff --git a/README.md b/README.md index f66ac8de4..c98751418 100644 --- a/README.md +++ b/README.md @@ -30,15 +30,14 @@ At runtime, fish requires: * a number of common UNIX utilities: coreutils (at least ls, seq, rm, mktemp), hostname, grep, awk, sed, and getopt. * bc, the "basic calculator" program. -Optional functionality should degrade gracefully. If fish ever threw an error about missing dependencies for an optional feature, it would be a bug in fish and should be reported on our GitHub. Optional features and dependencies include: +Optional functionality should degrade gracefully. (If fish ever threw a syntax error or a core dump due to missing dependencies for an optional feature, it would be a bug in fish and should be reported on our GitHub.) Optional features and dependencies include: -* fish's builtin `test` syntax, which requires `diff` and `expect`. You don't need these, though, if you don't want to use our tests. * dynamically generating usage tips for builtin functions, which requires man, apropos, nroff, and ul. * manual page parsing for auto-completion for external commands, which also requires the above man tools, plus Python 2.6+ or Python 3.3+. Python 2 will also need the backports.lzma package; Python 3.3+ should include the required module by default. * the web configuration tool, which requires Python 2.6+ or 3.3+. * the fish_clipboard_* functions (bound to \cv and \cx), which require xsel or pbcopy/pbpaste. -fish can also show VCS information in the prompt, when the current working directory is in a VCS-tracked project. This is one of the weakest dependencies, though: if you don't use git, you don't need git information in the prompt, and if you do use git, you already have it. +fish can also show VCS information in the prompt, when the current working directory is in a VCS-tracked project, with support for Git, Mercurial, and Subversion. This is one of the weakest dependencies, though: if you don't use git, for example, you don't need git information in the prompt, and if you do use git, you already have it. Similarly, fish comes with a variety of auto-completion scripts to help with command-line usage of a variety of applications; these scripts call the application in question, but you should only need these applications if you use them. ### Autotools Build From e27855b22569523a8f9ce4f417d90a0788acf1bd Mon Sep 17 00:00:00 2001 From: Andrew Toskin Date: Fri, 11 Aug 2017 03:52:38 -0700 Subject: [PATCH 11/79] Clarify dependencies: required vs optional, and build vs runtime. (#4301) * Clarify dependencies: required vs optional, and build vs runtime. A first pass at updating the dependency documentation, based on the discussion in this thread: https://github.com/fish-shell/fish-shell/issues/2062 * Clarify notes on dependency errors, tests, and VCS integration. An optional feature that suggests you install Python is okay; core-dumping is not. The note on tests was about fish development tests, not the `test` builtin for conditional syntax. Specifically mention git, hg, and svn in the VCS section. --- CONTRIBUTING.md | 2 ++ README.md | 42 ++++++++++++++++++++---------------------- 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 373c3dc9a..a01431df2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -31,6 +31,8 @@ To make linting the code easy there are two make targets: `lint` and `lint-all`. Fish has custom cppcheck rules in the file `.cppcheck.rule`. These help catch mistakes such as using `wcwidth()` rather than `fish_wcwidth()`. Please add a new rule if you find similar mistakes being made. +Fish also depends on `diff` and `expect` for its tests. + ### Dealing With Lint Warnings You are strongly encouraged to address a lint warning by refactoring the code, changing variable names, or whatever action is implied by the warning. diff --git a/README.md b/README.md index 24993bc81..c98751418 100644 --- a/README.md +++ b/README.md @@ -13,17 +13,31 @@ Detailed user documentation is available by running `help` within fish, and also ## Building -fish requires a C++11 compiler. It builds successfully with g++ 4.8 or later, or with clang 3.3 or later. +### Dependencies -fish can be built using autotools or Xcode. autoconf 2.60 or later, as well as automake 1.13 or later, are required to build from git versions. These are not required to build from released tarballs. +Compiling fish requires: -fish depends on a curses implementation, such as ncurses. The headers and libraries are required for building. +* a C++11 compiler. It builds successfully with g++ 4.8 or later, or with clang 3.3 or later. fish can be built using autotools or Xcode. autoconf 2.60 or later, as well as automake 1.13 or later, are required to build from git versions, but these are not required to build from the officially released tarballs. +* the headers and libraries of a curses implementation such as ncurses. +* gettext, for translation support. +* Doxygen 1.8.7 or newer, for building the documentation. -fish requires PCRE2 due to the regular expression support contained in the `string` builtin. A copy is included with the source code, and will be used automatically if it does not already exist on your system. +At runtime, fish requires: -fish requires gettext for translation support. +* a curses implementation, such as ncurses. +* gettext, if you need the localizations. +* PCRE2, due to the regular expression support contained in the `string` builtin. A copy is included with the source code, and will be used automatically if it does not already exist on your system. +* a number of common UNIX utilities: coreutils (at least ls, seq, rm, mktemp), hostname, grep, awk, sed, and getopt. +* bc, the "basic calculator" program. -Building the documentation requires Doxygen 1.8.7 or newer. +Optional functionality should degrade gracefully. (If fish ever threw a syntax error or a core dump due to missing dependencies for an optional feature, it would be a bug in fish and should be reported on our GitHub.) Optional features and dependencies include: + +* dynamically generating usage tips for builtin functions, which requires man, apropos, nroff, and ul. +* manual page parsing for auto-completion for external commands, which also requires the above man tools, plus Python 2.6+ or Python 3.3+. Python 2 will also need the backports.lzma package; Python 3.3+ should include the required module by default. +* the web configuration tool, which requires Python 2.6+ or 3.3+. +* the fish_clipboard_* functions (bound to \cv and \cx), which require xsel or pbcopy/pbpaste. + +fish can also show VCS information in the prompt, when the current working directory is in a VCS-tracked project, with support for Git, Mercurial, and Subversion. This is one of the weakest dependencies, though: if you don't use git, for example, you don't need git information in the prompt, and if you do use git, you already have it. Similarly, fish comes with a variety of auto-completion scripts to help with command-line usage of a variety of applications; these scripts call the application in question, but you should only need these applications if you use them. ### Autotools Build @@ -55,22 +69,6 @@ On RedHat, CentOS, or Amazon EC2: sudo yum install ncurses-devel -## Runtime Dependencies - -fish requires a curses implementation, such as ncurses, to run. - -fish requires PCRE2 due to the regular expression support contained in the `string` builtin. A bundled version will be compiled in automatically at build time if required. - -fish requires a number of utilities to operate, which should be present on any Unix, GNU/Linux or OS X system. These include (but are not limited to) hostname, grep, awk, sed, and getopt. fish also requires the bc program. - -Translation support requires the gettext program. - -Usage output for builtin functions is generated on-demand from the installed manpages using `nroff` and `ul`. - -Some optional features of fish, such as the manual page completion parser and the web configuration tool, require Python. - -In order to generate completions from man pages compressed with either lzma or xz, you may need to install an extra Python package. Python versions prior to 2.6 are not supported. To process lzma-compresed manpages, backports.lzma is needed for Python 3.2 or older. From version 3.3 onwards, Python already includes the required module. - ## Packages for Linux Instructions on how to find builds for several Linux distros are at From 4dbb274d8af6991200939797abe62247933485bc Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 11 Aug 2017 15:05:28 +0200 Subject: [PATCH 12/79] Clarify docs on $status with and/or/begin/end See #4311. --- doc_src/and.txt | 2 +- doc_src/begin.txt | 2 +- doc_src/end.txt | 2 +- doc_src/or.txt | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/doc_src/and.txt b/doc_src/and.txt index ed56da62d..2670a35e1 100644 --- a/doc_src/and.txt +++ b/doc_src/and.txt @@ -11,7 +11,7 @@ COMMAND1; and COMMAND2 `and` statements may be used as part of the condition in an `if` or `while` block. See the documentation for `if` and `while` for examples. -`and` does not change the current exit status. The exit status of the last foreground command to exit can always be accessed using the $status variable. +`and` does not change the current exit status itself, but the command it runs most likely will. The exit status of the last foreground command to exit can always be accessed using the $status variable. \subsection and-example Example diff --git a/doc_src/begin.txt b/doc_src/begin.txt index 6d18a6061..374fc78ce 100644 --- a/doc_src/begin.txt +++ b/doc_src/begin.txt @@ -13,7 +13,7 @@ The block is unconditionally executed. `begin; ...; end` is equivalent to `if tr `begin` is used to group a number of commands into a block. This allows the introduction of a new variable scope, redirection of the input or output of a set of commands as a group, or to specify precedence when using the conditional commands like `and`. -`begin` does not change the current exit status. +`begin` does not change the current exit status itself. After the block has completed, `$status` will be set to the status returned by the most recent command. \subsection begin-example Example diff --git a/doc_src/end.txt b/doc_src/end.txt index 76cbe3603..2b035930c 100644 --- a/doc_src/end.txt +++ b/doc_src/end.txt @@ -16,4 +16,4 @@ switch VALUE; [case [WILDCARD...]; [COMMANDS...]; ...] end For more information, read the documentation for the block constructs, such as `if`, `for` and `while`. -The `end` command does not change the current exit status. +The `end` command does not change the current exit status. Instead, the status after it will be the status returned by the most recent command. diff --git a/doc_src/or.txt b/doc_src/or.txt index 97707e064..40b6bfcc7 100644 --- a/doc_src/or.txt +++ b/doc_src/or.txt @@ -12,7 +12,7 @@ COMMAND1; or COMMAND2 `or` statements may be used as part of the condition in an `and` or `while` block. See the documentation for `if` and `while` for examples. -`or` does not change the current exit status. The exit status of the last foreground command to exit can always be accessed using the $status variable. +`or` does not change the current exit status itself, but the command it runs most likely will. The exit status of the last foreground command to exit can always be accessed using the $status variable. \subsection or-example Example From 219c7ec81212752d755a33beccb891f863e427b7 Mon Sep 17 00:00:00 2001 From: Fabian Homborg Date: Fri, 11 Aug 2017 18:27:58 +0200 Subject: [PATCH 13/79] help: Fix error if no argument is given --- share/functions/help.fish | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/share/functions/help.fish b/share/functions/help.fish index a31d6af04..d8c4f26c6 100644 --- a/share/functions/help.fish +++ b/share/functions/help.fish @@ -100,7 +100,7 @@ function help --description 'Show help for the fish shell' case "*" # If $fish_help_item is empty, this will fail, # and $fish_help_page will end up as index.html - if type -q -f $fish_help_item + if type -q -f "$fish_help_item" # Prefer to use fish's man pages, to avoid # the annoying useless "builtin" man page bash # installs on OS X From d09d2ca9ca4d37790ac0d25f5cd7fe315cf2607a Mon Sep 17 00:00:00 2001 From: Mahmoud Al-Qudsi Date: Fri, 11 Aug 2017 13:51:28 -0500 Subject: [PATCH 14/79] Ignore more invalid arguments from parsed man pages Specifically closes #4313. Not being as agressive in what we ignore/blacklist, but can be revisited easily in the future to add more characters to the argument blacklist. --- share/tools/create_manpage_completions.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/share/tools/create_manpage_completions.py b/share/tools/create_manpage_completions.py index 0eb4045bc..598cff0a2 100755 --- a/share/tools/create_manpage_completions.py +++ b/share/tools/create_manpage_completions.py @@ -131,11 +131,12 @@ def built_command(options, description): fish_options = [] for optionstr in man_optionlist: option = re.sub(r"(\[.*\])", "", optionstr) - option = option.strip(" \t\n[]()") + option = option.strip(" \t\r\n[](){}.,:!") # Skip some problematic cases if option in ['-', '--']: continue + if any(c in "{}()" for c in option): continue if option.startswith('--'): # New style long option (--recursive) From 67f7e53237c280f16b54ace4061486d3aefc30ba Mon Sep 17 00:00:00 2001 From: Mahmoud Al-Qudsi Date: Fri, 11 Aug 2017 13:51:28 -0500 Subject: [PATCH 15/79] Ignore more invalid arguments from parsed man pages Specifically closes #4313. Not being as agressive in what we ignore/blacklist, but can be revisited easily in the future to add more characters to the argument blacklist. --- share/tools/create_manpage_completions.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/share/tools/create_manpage_completions.py b/share/tools/create_manpage_completions.py index 0eb4045bc..598cff0a2 100755 --- a/share/tools/create_manpage_completions.py +++ b/share/tools/create_manpage_completions.py @@ -131,11 +131,12 @@ def built_command(options, description): fish_options = [] for optionstr in man_optionlist: option = re.sub(r"(\[.*\])", "", optionstr) - option = option.strip(" \t\n[]()") + option = option.strip(" \t\r\n[](){}.,:!") # Skip some problematic cases if option in ['-', '--']: continue + if any(c in "{}()" for c in option): continue if option.startswith('--'): # New style long option (--recursive) From 6e7956a41383b3273981916901c6db091ea0c6ed Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Thu, 10 Aug 2017 17:46:08 -0700 Subject: [PATCH 16/79] change order of `env_set()` args It's bugged me forever that the scope is the second arg to `env_get()` but not `env_set()`. And since I'll be introducing some helper functions that wrap `env_set()` now is a good time to change the order of its arguments. --- src/builtin_argparse.cpp | 14 ++++++------- src/builtin_fg.cpp | 2 +- src/builtin_read.cpp | 22 ++++++++++---------- src/builtin_set.cpp | 10 ++++----- src/common.cpp | 4 ++-- src/env.cpp | 44 ++++++++++++++++++++-------------------- src/env.h | 2 +- src/exec.cpp | 2 +- src/fish.cpp | 2 +- src/fish_tests.cpp | 6 +++--- src/function.cpp | 4 ++-- src/input.cpp | 2 +- src/parse_execution.cpp | 2 +- src/path.cpp | 2 +- src/reader.cpp | 12 +++++------ 15 files changed, 65 insertions(+), 65 deletions(-) diff --git a/src/builtin_argparse.cpp b/src/builtin_argparse.cpp index 55e7afc4d..f1d934b65 100644 --- a/src/builtin_argparse.cpp +++ b/src/builtin_argparse.cpp @@ -458,13 +458,13 @@ static int validate_arg(argparse_cmd_opts_t &opts, option_spec_t *opt_spec, bool wcstring_list_t cmd_output; env_push(true); - env_set(L"_argparse_cmd", opts.name.c_str(), ENV_LOCAL); + env_set(L"_argparse_cmd", ENV_LOCAL, opts.name.c_str()); if (is_long_flag) { - env_set(var_name_prefix + L"name", opt_spec->long_flag.c_str(), ENV_LOCAL); + env_set(var_name_prefix + L"name", ENV_LOCAL, opt_spec->long_flag.c_str()); } else { - env_set(var_name_prefix + L"name", wcstring(1, opt_spec->short_flag).c_str(), ENV_LOCAL); + env_set(var_name_prefix + L"name", ENV_LOCAL, wcstring(1, opt_spec->short_flag).c_str()); } - env_set(var_name_prefix + L"value", woptarg, ENV_LOCAL); + env_set(var_name_prefix + L"value", ENV_LOCAL, woptarg); int retval = exec_subshell(opt_spec->validation_command, cmd_output, false); for (auto it : cmd_output) { @@ -645,7 +645,7 @@ static void set_argparse_result_vars(argparse_cmd_opts_t &opts) { auto val = list_to_array_val(opt_spec->vals); if (opt_spec->short_flag_valid) { - env_set(var_name_prefix + opt_spec->short_flag, val->c_str(), ENV_LOCAL); + env_set(var_name_prefix + opt_spec->short_flag, ENV_LOCAL, val->c_str()); } if (!opt_spec->long_flag.empty()) { // We do a simple replacement of all non alphanum chars rather than calling @@ -654,12 +654,12 @@ static void set_argparse_result_vars(argparse_cmd_opts_t &opts) { for (size_t pos = 0; pos < long_flag.size(); pos++) { if (!iswalnum(long_flag[pos])) long_flag[pos] = L'_'; } - env_set(var_name_prefix + long_flag, val->c_str(), ENV_LOCAL); + env_set(var_name_prefix + long_flag, ENV_LOCAL, val->c_str()); } } auto val = list_to_array_val(opts.argv); - env_set(L"argv", val->c_str(), ENV_LOCAL); + env_set(L"argv", ENV_LOCAL, val->c_str()); } /// The argparse builtin. This is explicitly not compatible with the BSD or GNU version of this diff --git a/src/builtin_fg.cpp b/src/builtin_fg.cpp index 4fb05c846..33255cf71 100644 --- a/src/builtin_fg.cpp +++ b/src/builtin_fg.cpp @@ -102,7 +102,7 @@ int builtin_fg(parser_t &parser, io_streams_t &streams, wchar_t **argv) { } const wcstring ft = tok_first(j->command()); - if (!ft.empty()) env_set(L"_", ft.c_str(), ENV_EXPORT); + if (!ft.empty()) env_set(L"_", ENV_EXPORT, ft.c_str()); reader_write_title(j->command()); job_promote(j); diff --git a/src/builtin_read.cpp b/src/builtin_read.cpp index 7d6a0522a..6e50e3a7f 100644 --- a/src/builtin_read.cpp +++ b/src/builtin_read.cpp @@ -422,7 +422,7 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { if (exit_res != STATUS_CMD_OK) { // Define the var(s) without any data. We do this because when this happens we want the user // to be able to use the var but have it expand to nothing. - for (int i = 0; i < argc; i++) env_set(argv[i], NULL, opts.place); + for (int i = 0; i < argc; i++) env_set(argv[i], opts.place, NULL); return exit_res; } @@ -444,9 +444,9 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { *out = *it; out += 2; } - env_set(argv[0], chars.c_str(), opts.place); + env_set(argv[0], opts.place, chars.c_str()); } else { - env_set(argv[0], NULL, opts.place); + env_set(argv[0], opts.place, NULL); } } else { // not array mode int i = 0; @@ -454,12 +454,12 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { for (; i + 1 < argc; ++i) { if (j < bufflen) { wchar_t buffer[2] = {buff[j++], 0}; - env_set(argv[i], buffer, opts.place); + env_set(argv[i], opts.place, buffer); } else { - env_set(argv[i], L"", opts.place); + env_set(argv[i], opts.place, L""); } } - if (i < argc) env_set(argv[i], &buff[j], opts.place); + if (i < argc) env_set(argv[i], opts.place, &buff[j]); } } else if (opts.array) { if (!opts.have_delimiter) { @@ -473,21 +473,21 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { if (!tokens.empty()) tokens.push_back(ARRAY_SEP); tokens.append(buff, loc.first, loc.second); } - env_set(argv[0], tokens.empty() ? NULL : tokens.c_str(), opts.place); + env_set(argv[0], opts.place, tokens.empty() ? NULL : tokens.c_str()); } else { wcstring_list_t splits; split_about(buff.begin(), buff.end(), opts.delimiter.begin(), opts.delimiter.end(), &splits, LONG_MAX); auto val = list_to_array_val(splits); - env_set(argv[0], val->c_str(), opts.place); + env_set(argv[0], opts.place, val->c_str()); } } else { // not array if (!opts.have_delimiter) { wcstring_range loc = wcstring_range(0, 0); for (int i = 0; i < argc; i++) { loc = wcstring_tok(buff, (i + 1 < argc) ? opts.delimiter : wcstring(), loc); - env_set(argv[i], loc.first == wcstring::npos ? L"" : &buff.c_str()[loc.first], - opts.place); + env_set(argv[i], opts.place, + loc.first == wcstring::npos ? L"" : &buff.c_str()[loc.first]); } } else { wcstring_list_t splits; @@ -496,7 +496,7 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { split_about(buff.begin(), buff.end(), opts.delimiter.begin(), opts.delimiter.end(), &splits, argc - 1); for (size_t i = 0; i < (size_t)argc && i < splits.size(); i++) { - env_set(argv[i], splits[i].c_str(), opts.place); + env_set(argv[i], opts.place, splits[i].c_str()); } } } diff --git a/src/builtin_set.cpp b/src/builtin_set.cpp index 8672e6fcc..41ab75aeb 100644 --- a/src/builtin_set.cpp +++ b/src/builtin_set.cpp @@ -284,8 +284,8 @@ static int my_env_path_setup(const wchar_t *cmd, const wchar_t *key, //!OCLINT( /// Call env_set. If this is a path variable, e.g. PATH, validate the elements. On error, print a /// description of the problem to stderr. -static int my_env_set(const wchar_t *cmd, const wchar_t *key, const wcstring_list_t &list, - int scope, io_streams_t &streams) { +static int my_env_set(const wchar_t *cmd, const wchar_t *key, int scope, + const wcstring_list_t &list, io_streams_t &streams) { int retval; if (is_path_variable(key)) { @@ -294,7 +294,7 @@ static int my_env_set(const wchar_t *cmd, const wchar_t *key, const wcstring_lis } auto val = list_to_array_val(list); - retval = env_set(key, val->c_str(), scope | ENV_USER); + retval = env_set(key, scope | ENV_USER, val->c_str()); switch (retval) { case ENV_OK: { retval = STATUS_CMD_OK; @@ -633,7 +633,7 @@ static int builtin_set_erase(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, wcstring_list_t result; dest_var.to_list(result); erase_values(result, indexes); - retval = my_env_set(cmd, dest, result, scope, streams); + retval = my_env_set(cmd, dest, scope, result, streams); } if (retval != STATUS_CMD_OK) return retval; @@ -742,7 +742,7 @@ static int builtin_set_set(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, w } if (retval != STATUS_CMD_OK) return retval; - retval = my_env_set(cmd, varname, new_values, scope, streams); + retval = my_env_set(cmd, varname, scope, new_values, streams); if (retval != STATUS_CMD_OK) return retval; return check_global_scope_exists(cmd, opts, varname, streams); } diff --git a/src/common.cpp b/src/common.cpp index 68cd2274d..9be4b1db9 100644 --- a/src/common.cpp +++ b/src/common.cpp @@ -1583,11 +1583,11 @@ static void export_new_termsize(struct winsize *new_termsize) { wchar_t buf[64]; env_var_t cols = env_get(L"COLUMNS", ENV_EXPORT); swprintf(buf, 64, L"%d", (int)new_termsize->ws_col); - env_set(L"COLUMNS", buf, ENV_GLOBAL | (cols.missing_or_empty() ? 0 : ENV_EXPORT)); + env_set(L"COLUMNS", ENV_GLOBAL | (cols.missing_or_empty() ? 0 : ENV_EXPORT), buf); env_var_t lines = env_get(L"LINES", ENV_EXPORT); swprintf(buf, 64, L"%d", (int)new_termsize->ws_row); - env_set(L"LINES", buf, ENV_GLOBAL | (lines.missing_or_empty() ? 0 : ENV_EXPORT)); + env_set(L"LINES", ENV_GLOBAL | (lines.missing_or_empty() ? 0 : ENV_EXPORT), buf); #ifdef HAVE_WINSIZE ioctl(STDOUT_FILENO, TIOCSWINSZ, new_termsize); diff --git a/src/env.cpp b/src/env.cpp index 1fa680d6b..c5d895bf9 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -369,7 +369,7 @@ static void fix_colon_delimited_var(const wcstring &var_name) { } } - int scope = env_set(var_name, new_val.c_str(), ENV_USER); + int scope = env_set(var_name, ENV_USER, new_val.c_str()); if (scope != ENV_OK) { debug(0, L"fix_colon_delimited_var failed unexpectedly with retval %d", scope); } @@ -667,7 +667,7 @@ static void setup_path() { const env_var_t path = env_get(L"PATH"); if (path.missing_or_empty()) { const wchar_t *value = L"/usr/bin" ARRAY_SEP_STR L"/bin"; - env_set(L"PATH", value, ENV_GLOBAL | ENV_EXPORT); + env_set(L"PATH", ENV_GLOBAL | ENV_EXPORT, value); } } @@ -676,10 +676,10 @@ static void setup_path() { /// adjusted. static void env_set_termsize() { env_var_t cols = env_get(L"COLUMNS"); - if (cols.missing_or_empty()) env_set(L"COLUMNS", DFLT_TERM_COL_STR, ENV_GLOBAL); + if (cols.missing_or_empty()) env_set(L"COLUMNS", ENV_GLOBAL, DFLT_TERM_COL_STR); env_var_t rows = env_get(L"LINES"); - if (rows.missing_or_empty()) env_set(L"LINES", DFLT_TERM_ROW_STR, ENV_GLOBAL); + if (rows.missing_or_empty()) env_set(L"LINES", ENV_GLOBAL, DFLT_TERM_ROW_STR); } bool env_set_pwd() { @@ -689,7 +689,7 @@ bool env_set_pwd() { _(L"Could not determine current working directory. Is your locale set correctly?")); return false; } - env_set(L"PWD", res.c_str(), ENV_EXPORT | ENV_GLOBAL); + env_set(L"PWD", ENV_EXPORT | ENV_GLOBAL, res.c_str()); return true; } @@ -728,7 +728,7 @@ static void setup_user(bool force) { int retval = getpwuid_r(getuid(), &userinfo, buf, sizeof(buf), &result); if (!retval && result) { const wcstring uname = str2wcstring(userinfo.pw_name); - env_set(L"USER", uname.c_str(), ENV_GLOBAL | ENV_EXPORT); + env_set(L"USER", ENV_GLOBAL | ENV_EXPORT, uname.c_str()); } } } @@ -820,7 +820,7 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { if (eql == wcstring::npos) { // No equals found. if (is_read_only(key_and_val) || is_electric(key_and_val)) continue; - env_set(key_and_val, L"", ENV_EXPORT | ENV_GLOBAL); + env_set(key_and_val, ENV_EXPORT | ENV_GLOBAL, L""); } else { key.assign(key_and_val, 0, eql); if (is_read_only(key) || is_electric(key)) continue; @@ -829,16 +829,16 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { std::replace(val.begin(), val.end(), L':', ARRAY_SEP); } - env_set(key, val.c_str(), ENV_EXPORT | ENV_GLOBAL); + env_set(key, ENV_EXPORT | ENV_GLOBAL, val.c_str()); } } // Set the given paths in the environment, if we have any. if (paths != NULL) { - env_set(FISH_DATADIR_VAR, paths->data.c_str(), ENV_GLOBAL); - env_set(FISH_SYSCONFDIR_VAR, paths->sysconf.c_str(), ENV_GLOBAL); - env_set(FISH_HELPDIR_VAR, paths->doc.c_str(), ENV_GLOBAL); - env_set(FISH_BIN_DIR, paths->bin.c_str(), ENV_GLOBAL); + env_set(FISH_DATADIR_VAR, ENV_GLOBAL, paths->data.c_str()); + env_set(FISH_SYSCONFDIR_VAR, ENV_GLOBAL, paths->sysconf.c_str()); + env_set(FISH_HELPDIR_VAR, ENV_GLOBAL, paths->doc.c_str()); + env_set(FISH_BIN_DIR, ENV_GLOBAL, paths->bin.c_str()); } init_locale(); @@ -859,7 +859,7 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { // Set up the version variable. wcstring version = str2wcstring(get_fish_version()); - env_set(L"FISH_VERSION", version.c_str(), ENV_GLOBAL); + env_set(L"FISH_VERSION", ENV_GLOBAL, version.c_str()); // Set up SHLVL variable. const env_var_t shlvl_var = env_get(L"SHLVL"); @@ -872,7 +872,7 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { nshlvl_str = to_string(shlvl_i + 1); } } - env_set(L"SHLVL", nshlvl_str.c_str(), ENV_GLOBAL | ENV_EXPORT); + env_set(L"SHLVL", ENV_GLOBAL | ENV_EXPORT, nshlvl_str.c_str()); env_read_only.insert(L"SHLVL"); // Set up the HOME variable. @@ -899,18 +899,18 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { } if (!retval && result && userinfo.pw_dir) { const wcstring dir = str2wcstring(userinfo.pw_dir); - env_set(L"HOME", dir.c_str(), ENV_GLOBAL | ENV_EXPORT); + env_set(L"HOME", ENV_GLOBAL | ENV_EXPORT, dir.c_str()); } else { // We cannot get $HOME, set it to the empty list. // This triggers warnings for history and config.fish already, // so it isn't necessary to warn here as well. - env_set(L"HOME", ENV_NULL, ENV_GLOBAL | ENV_EXPORT); + env_set(L"HOME", ENV_GLOBAL | ENV_EXPORT, ENV_NULL); } free(unam_narrow); } else { // If $USER is empty as well (which we tried to set above), // we can't get $HOME. - env_set(L"HOME", ENV_NULL, ENV_GLOBAL | ENV_EXPORT); + env_set(L"HOME", ENV_GLOBAL | ENV_EXPORT, ENV_NULL); } } @@ -924,7 +924,7 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { use_posix_spawn.missing_or_empty() ? true : from_string(use_posix_spawn.as_string()); // Set fish_bind_mode to "default". - env_set(FISH_BIND_MODE_VAR, DEFAULT_BIND_MODE, ENV_GLOBAL); + env_set(FISH_BIND_MODE_VAR, ENV_GLOBAL, DEFAULT_BIND_MODE); // This is somewhat subtle. At this point we consider our environment to be sufficiently // initialized that we can react to changes to variables. Prior to doing this we expect that the @@ -976,7 +976,7 @@ static env_node_t *env_get_node(const wcstring &key) { /// * ENV_SCOPE, the variable cannot be set in the given scope. This applies to readonly/electric /// variables set from the local or universal scopes, or set as exported. /// * ENV_INVALID, the variable value was invalid. This applies only to special variables. -int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t var_mode) { +int env_set(const wcstring &key, env_mode_flags_t var_mode, const wchar_t *val) { ASSERT_IS_MAIN_THREAD(); bool has_changed_old = vars_stack().has_changed_exported; int done = 0; @@ -986,7 +986,7 @@ int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t var_mode) wcstring val_canonical = val; path_make_canonical(val_canonical); if (val != val_canonical) { - return env_set(key, val_canonical.c_str(), var_mode); + return env_set(key, var_mode, val_canonical.c_str()); } } @@ -1528,9 +1528,9 @@ void env_set_argv(const wchar_t *const *argv) { sb.append(*arg); } - env_set(L"argv", sb.c_str(), ENV_LOCAL); + env_set(L"argv", ENV_LOCAL, sb.c_str()); } else { - env_set(L"argv", NULL, ENV_LOCAL); + env_set(L"argv", ENV_LOCAL, NULL); } } diff --git a/src/env.h b/src/env.h index 5c5890991..23a602bee 100644 --- a/src/env.h +++ b/src/env.h @@ -127,7 +127,7 @@ class env_var_t { /// \param mode An optional scope to search in. All scopes are searched if unset env_var_t env_get(const wcstring &key, env_mode_flags_t mode = ENV_DEFAULT); -int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t mode); +int env_set(const wcstring &key, env_mode_flags_t mode, const wchar_t *val); /// Returns true if the specified key exists. This can't be reliably done using env_get, since /// env_get returns null for 0-element arrays. diff --git a/src/exec.cpp b/src/exec.cpp index dc452d3b3..aa82f1bf0 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -384,7 +384,7 @@ void internal_exec(job_t *j, const io_chain_t &&all_ios) { shlvl_str = to_string(shlvl - 1); } } - env_set(L"SHLVL", shlvl_str.c_str(), ENV_GLOBAL | ENV_EXPORT); + env_set(L"SHLVL", ENV_GLOBAL | ENV_EXPORT, shlvl_str.c_str()); // launch_process _never_ returns. launch_process_nofork(j->processes.front().get()); diff --git a/src/fish.cpp b/src/fish.cpp index a3f45dfe3..1a45cb9f4 100644 --- a/src/fish.cpp +++ b/src/fish.cpp @@ -421,7 +421,7 @@ int main(int argc, char **argv) { list.push_back(str2wcstring(*ptr)); } auto val = list_to_array_val(list); - env_set(L"argv", val->c_str(), ENV_DEFAULT); + env_set(L"argv", ENV_DEFAULT, val->c_str()); const wcstring rel_filename = str2wcstring(file); diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 9c539d96e..822d71b99 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -1617,7 +1617,7 @@ static void test_abbreviations(void) { {L"gc", L"git checkout"}, {L"foo", L"bar"}, {L"gx", L"git checkout"}, }; for (auto it : abbreviations) { - int ret = env_set(L"_fish_abbr_" + it.first, it.second.c_str(), ENV_LOCAL); + int ret = env_set(L"_fish_abbr_" + it.first, ENV_LOCAL, it.second.c_str()); if (ret != 0) err(L"Unable to set abbreviation variable"); } @@ -2479,7 +2479,7 @@ static void test_autosuggest_suggest_special() { perform_one_autosuggestion_cd_test(L"cd 'test/autosuggest_test/5", vars, L"foo\"bar/", __LINE__); - env_set(L"AUTOSUGGEST_TEST_LOC", wd.c_str(), ENV_LOCAL); + env_set(L"AUTOSUGGEST_TEST_LOC", ENV_LOCAL, wd.c_str()); perform_one_autosuggestion_cd_test(L"cd $AUTOSUGGEST_TEST_LOC/0", vars, L"foobar/", __LINE__); perform_one_autosuggestion_cd_test(L"cd ~/test_autosuggest_suggest_specia", vars, L"l/", __LINE__); @@ -4255,7 +4255,7 @@ long return_timezone_hour(time_t tstamp, const wchar_t *timezone) { char *str_ptr; size_t n; - env_set(L"TZ", timezone, ENV_EXPORT); + env_set(L"TZ", ENV_EXPORT, timezone); localtime_r(&tstamp, <ime); n = strftime(ltime_str, 3, "%H", <ime); if (n != 2) { diff --git a/src/function.cpp b/src/function.cpp index 44105c130..9dd3d8264 100644 --- a/src/function.cpp +++ b/src/function.cpp @@ -340,7 +340,7 @@ void function_prepare_environment(const wcstring &name, const wchar_t *const *ar const wchar_t *const *arg; size_t i; for (i = 0, arg = argv; i < named_arguments.size(); i++) { - env_set(named_arguments.at(i).c_str(), *arg, ENV_LOCAL | ENV_USER); + env_set(named_arguments.at(i).c_str(), ENV_LOCAL | ENV_USER, *arg); if (*arg) arg++; } @@ -349,6 +349,6 @@ void function_prepare_environment(const wcstring &name, const wchar_t *const *ar for (std::map::const_iterator it = inherited_vars.begin(), end = inherited_vars.end(); it != end; ++it) { - env_set(it->first, it->second.missing() ? NULL : it->second.c_str(), ENV_LOCAL | ENV_USER); + env_set(it->first, ENV_LOCAL | ENV_USER, it->second.missing() ? NULL : it->second.c_str()); } } diff --git a/src/input.cpp b/src/input.cpp index 94d4a4469..456e07766 100644 --- a/src/input.cpp +++ b/src/input.cpp @@ -211,7 +211,7 @@ void input_set_bind_mode(const wcstring &bm) { // modes may not be empty - empty is a sentinel value meaning to not change the mode assert(!bm.empty()); if (input_get_bind_mode() != bm.c_str()) { - env_set(FISH_BIND_MODE_VAR, bm.c_str(), ENV_GLOBAL); + env_set(FISH_BIND_MODE_VAR, ENV_GLOBAL, bm.c_str()); } } diff --git a/src/parse_execution.cpp b/src/parse_execution.cpp index f21327569..1b1557711 100644 --- a/src/parse_execution.cpp +++ b/src/parse_execution.cpp @@ -473,7 +473,7 @@ parse_execution_result_t parse_execution_context_t::run_for_statement( } const wcstring &val = argument_sequence.at(i); - env_set(for_var_name, val.c_str(), ENV_LOCAL); + env_set(for_var_name, ENV_LOCAL, val.c_str()); fb->loop_status = LOOP_NORMAL; this->run_job_list(block_contents, fb); diff --git a/src/path.cpp b/src/path.cpp index e2c7216af..f57bb0f8c 100644 --- a/src/path.cpp +++ b/src/path.cpp @@ -260,7 +260,7 @@ static void maybe_issue_path_warning(const wcstring &which_dir, const wcstring & if (env_exist(warning_var_name.c_str(), ENV_GLOBAL | ENV_EXPORT)) { return; } - env_set(warning_var_name, L"1", ENV_GLOBAL | ENV_EXPORT); + env_set(warning_var_name, ENV_GLOBAL | ENV_EXPORT, L"1"); debug(0, custom_error_msg.c_str()); if (path.empty()) { diff --git a/src/reader.cpp b/src/reader.cpp index cd7491e44..7ad722e72 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -765,8 +765,8 @@ void reader_init() { // Ensure this var is present even before an interactive command is run so that if it is used // in a function like `fish_prompt` or `fish_right_prompt` it is defined at the time the first - // prompt is issued. - env_set(ENV_CMD_DURATION, L"0", ENV_UNEXPORT); + // prompt is written. + env_set(ENV_CMD_DURATION, ENV_UNEXPORT, L"0"); // Save the initial terminal mode. tcgetattr(STDIN_FILENO, &terminal_mode_on_startup); @@ -1650,7 +1650,7 @@ static void reader_interactive_init() { wperror(L"tcsetattr"); } - env_set(L"_", L"fish", ENV_GLOBAL); + env_set(L"_", ENV_GLOBAL, L"fish"); } /// Destroy data for interactive use. @@ -1917,7 +1917,7 @@ void set_env_cmd_duration(struct timeval *after, struct timeval *before) { } swprintf(buf, 16, L"%d", (secs * 1000) + (usecs / 1000)); - env_set(ENV_CMD_DURATION, buf, ENV_UNEXPORT); + env_set(ENV_CMD_DURATION, ENV_UNEXPORT, buf); } void reader_run_command(parser_t &parser, const wcstring &cmd) { @@ -1925,7 +1925,7 @@ void reader_run_command(parser_t &parser, const wcstring &cmd) { wcstring ft = tok_first(cmd); - if (!ft.empty()) env_set(L"_", ft.c_str(), ENV_GLOBAL); + if (!ft.empty()) env_set(L"_", ENV_GLOBAL, ft.c_str()); reader_write_title(cmd); @@ -1941,7 +1941,7 @@ void reader_run_command(parser_t &parser, const wcstring &cmd) { term_steal(); - env_set(L"_", program_name, ENV_GLOBAL); + env_set(L"_", ENV_GLOBAL, program_name); #ifdef HAVE__PROC_SELF_STAT proc_update_jiffies(); From 3df8643c31a4053aa04410d967886a5888adb1e0 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Thu, 10 Aug 2017 21:11:47 -0700 Subject: [PATCH 17/79] make `missing_var` a singleton Make the `env_var_t::missing_var()` object a singleton rather than a dynamically constructed object. This requires some discipline in its use since C++ doesn't directly support immutable objects. But it is slightly more efficient and helps identify code that incorrectly mutates `env_var_t` objects that should not be modified. --- src/common.cpp | 1 + src/env.cpp | 33 ++++++++++++++++++++++----------- src/env.h | 36 +++++++++++++++++------------------- src/env_universal_common.cpp | 10 ++++------ src/expand.cpp | 4 ++-- src/fish_tests.cpp | 2 +- src/highlight.cpp | 2 +- 7 files changed, 48 insertions(+), 40 deletions(-) diff --git a/src/common.cpp b/src/common.cpp index 9be4b1db9..2bec05227 100644 --- a/src/common.cpp +++ b/src/common.cpp @@ -1581,6 +1581,7 @@ static void validate_new_termsize(struct winsize *new_termsize) { /// Export the new terminal size as env vars and to the kernel if possible. static void export_new_termsize(struct winsize *new_termsize) { wchar_t buf[64]; + env_var_t cols = env_get(L"COLUMNS", ENV_EXPORT); swprintf(buf, 64, L"%d", (int)new_termsize->ws_col); env_set(L"COLUMNS", ENV_GLOBAL | (cols.missing_or_empty() ? 0 : ENV_EXPORT), buf); diff --git a/src/env.cpp b/src/env.cpp index c5d895bf9..5a4b1b0d9 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -107,6 +107,15 @@ static void init_locale(); static void init_curses(); static void tokenize_variable_array(const wcstring &val, wcstring_list_t &out); +/// Construct a missing var object +env_var_t create_missing_var() { + env_var_t var; + var.set_missing(); + return var; +} + +env_var_t missing_var = create_missing_var(); + // Struct representing one level in the function variable stack. // Only our variable stack should create and destroy these class env_node_t { @@ -306,7 +315,7 @@ static bool is_electric(const wcstring &key) { const env_var_t env_node_t::find_entry(const wcstring &key) { var_table_t::const_iterator entry = env.find(key); if (entry != env.end()) return entry->second; - return env_var_t::missing_var(); + return missing_var; } /// Return the current umask value. @@ -1209,7 +1218,7 @@ int env_remove(const wcstring &key, int var_mode) { } wcstring env_var_t::as_string(void) const { - //assert(!is_missing); + assert(!is_missing); return val; } @@ -1235,12 +1244,12 @@ env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { // Make the assumption that electric keys can't be shadowed elsewhere, since we currently block // that in env_set(). if (is_electric(key)) { - if (!search_global) return env_var_t::missing_var(); + if (!search_global) return missing_var; if (key == L"history") { // Big hack. We only allow getting the history on the main thread. Note that history_t // may ask for an environment variable, so don't take the lock here (we don't need it). if (!is_main_thread()) { - return env_var_t::missing_var(); + return missing_var; } history_t *history = reader_get_history(); @@ -1280,7 +1289,7 @@ env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { } } - if (!search_universal) return env_var_t::missing_var(); + if (!search_universal) return missing_var; // Another hack. Only do a universal barrier on the main thread (since it can change variable // values). Make sure we do this outside the env_lock because it may itself call `env_get()`. @@ -1289,15 +1298,17 @@ env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { env_universal_barrier(); } + // Okay, we couldn't find a local or global var given the requirements. If there is a matching + // universal var return that. if (uvars()) { env_var_t env_var = uvars()->get(key); - if (env_var == ENV_NULL || - !(uvars()->get_export(key) ? search_exported : search_unexported)) { - env_var = env_var_t::missing_var(); + if (!env_var.missing() && + (uvars()->get_export(key) ? search_exported : search_unexported)) { + return env_var; } - return env_var; } - return env_var_t::missing_var(); + + return missing_var; } bool env_exist(const wchar_t *key, env_mode_flags_t mode) { @@ -1561,7 +1572,7 @@ env_var_t env_vars_snapshot_t::get(const wcstring &key) const { return env_get(key); } std::map::const_iterator iter = vars.find(key); - return iter == vars.end() ? env_var_t::missing_var() : env_var_t(iter->second); + return iter == vars.end() ? missing_var : env_var_t(iter->second); } const wchar_t *const env_vars_snapshot_t::highlighting_keys[] = {L"PATH", L"CDPATH", diff --git a/src/env.h b/src/env.h index 23a602bee..e29dda483 100644 --- a/src/env.h +++ b/src/env.h @@ -77,19 +77,13 @@ class env_var_t { public: bool exportv; // whether the variable should be exported - static env_var_t missing_var() { - env_var_t result((wcstring())); - result.is_missing = true; - result.exportv = false; - return result; - } - - env_var_t(const env_var_t &x) : val(x.val), is_missing(x.is_missing), exportv(x.exportv) {} env_var_t(const wcstring &x) : val(x), is_missing(false), exportv(false) {} env_var_t(const wchar_t *x) : val(x), is_missing(false), exportv(false) {} env_var_t() : val(L""), is_missing(false), exportv(false) {} - bool empty(void) const { return val.empty() || val == ENV_NULL; }; + void set_missing(void) { is_missing = true; } + + bool empty(void) const { return val.empty() || val == ENV_NULL; } bool missing(void) const { return is_missing; } bool missing_or_empty(void) const { return missing() || empty(); } @@ -97,29 +91,33 @@ class env_var_t { void to_list(wcstring_list_t &out) const; wcstring as_string() const; - env_var_t &operator=(const env_var_t &v) { - is_missing = v.is_missing; - exportv = v.exportv; - val = v.val; - return *this; + // You cannot convert a missing var to a non-missing var. You can only change the value of a var + // that is not missing. + void set_val(const wcstring &s) { + assert(!is_missing); + val = s; + } + void set_val(const wchar_t *s) { + assert(!is_missing); + val = s; } - - void set_val(const wcstring &s) { val = s; is_missing = false; } - void set_val(const wchar_t *s) { val = s; is_missing = false; } bool operator==(const env_var_t &s) const { return is_missing == s.is_missing && val == s.val; } bool operator==(const wcstring &s) const { return !is_missing && val == s; } + bool operator==(const wchar_t *s) const { return !is_missing && val == s; } + bool operator!=(const env_var_t &v) const { return val != v.val; } bool operator!=(const wcstring &s) const { return val != s; } bool operator!=(const wchar_t *s) const { return val != s; } - - bool operator==(const wchar_t *s) const { return !is_missing && val == s; } }; +env_var_t create_missing_var(); +extern env_var_t missing_var; + /// Gets the variable with the specified name, or env_var_t::missing_var if it does not exist or is /// an empty array. /// diff --git a/src/env_universal_common.cpp b/src/env_universal_common.cpp index 19a087ee1..ee0bd80cb 100644 --- a/src/env_universal_common.cpp +++ b/src/env_universal_common.cpp @@ -270,12 +270,9 @@ env_universal_t::env_universal_t(const wcstring &path) env_universal_t::~env_universal_t() { pthread_mutex_destroy(&lock); } env_var_t env_universal_t::get(const wcstring &name) const { - env_var_t result = env_var_t::missing_var(); var_table_t::const_iterator where = vars.find(name); - if (where != vars.end()) { - result = where->second; - } - return result; + if (where != vars.end()) return where->second; + return missing_var; } bool env_universal_t::get_export(const wcstring &name) const { @@ -465,7 +462,8 @@ bool env_universal_t::write_to_fd(int fd, const wcstring &path) { // variable; soldier on. const wcstring &key = iter->first; const env_var_t &var = iter->second; - append_file_entry(var.exportv ? SET_EXPORT : SET, key, var.as_string(), &contents, &storage); + append_file_entry(var.exportv ? SET_EXPORT : SET, key, var.as_string(), &contents, + &storage); // Go to next. ++iter; diff --git a/src/expand.cpp b/src/expand.cpp index 2bcc6ec18..9de0a38d7 100644 --- a/src/expand.cpp +++ b/src/expand.cpp @@ -144,7 +144,7 @@ static void append_cmdsub_error(parse_error_list_t *errors, size_t source_start, /// Return the environment variable value for the string starting at \c in. static env_var_t expand_var(const wchar_t *in) { - if (!in) return env_var_t::missing_var(); + if (!in) return missing_var; return env_get(in); } @@ -777,7 +777,7 @@ static int expand_variables(const wcstring &instr, std::vector *ou var_tmp.append(instr, start_pos, var_len); env_var_t var; if (var_len == 1 && var_tmp[0] == VARIABLE_EXPAND_EMPTY) { - var = env_var_t::missing_var(); + var = missing_var; } else { var = expand_var(var_tmp.c_str()); } diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 822d71b99..c7e6c200f 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -2658,7 +2658,7 @@ static void test_universal() { const wcstring key = format_string(L"key_%d_%d", i, j); env_var_t expected_val; if (j == 0) { - expected_val = env_var_t::missing_var(); + expected_val = missing_var; } else { expected_val = format_string(L"val_%d_%d", i, j); } diff --git a/src/highlight.cpp b/src/highlight.cpp index aefafa436..7186db127 100644 --- a/src/highlight.cpp +++ b/src/highlight.cpp @@ -222,7 +222,7 @@ static bool is_potential_cd_path(const wcstring &path, const wcstring &working_d } else { // Get the CDPATH. env_var_t cdpath = env_get(L"CDPATH"); - if (cdpath.missing_or_empty()) cdpath = L"."; + if (cdpath.missing_or_empty()) cdpath = env_var_t(L"."); // Tokenize it into directories. std::vector pathsv; From 4cf480a1a5f08bde2f8a8d98941e6f45c0d70565 Mon Sep 17 00:00:00 2001 From: sentriz Date: Sat, 12 Aug 2017 00:09:13 +0100 Subject: [PATCH 18/79] Accept return as a valid answer to 'Edit the file again?' --- share/functions/funced.fish | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/share/functions/funced.fish b/share/functions/funced.fish index 687a6ffa1..a889a42c9 100644 --- a/share/functions/funced.fish +++ b/share/functions/funced.fish @@ -105,9 +105,7 @@ function funced --description 'Edit function definition' echo # add a line between the parse error and the prompt set -l repeat set -l prompt (_ 'Edit the file again\? [Y/n]') - while test -z "$repeat" - read -p "echo $prompt\ " repeat - end + read -p "echo $prompt\ " repeat if not contains $repeat n N no NO No nO continue end From 44ef6cc87f03ae35a16587daeba39a9c01f74e45 Mon Sep 17 00:00:00 2001 From: sentriz Date: Sat, 12 Aug 2017 01:53:54 +0100 Subject: [PATCH 19/79] Check for affirmative answer --- share/functions/funced.fish | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/share/functions/funced.fish b/share/functions/funced.fish index a889a42c9..e8e05a72a 100644 --- a/share/functions/funced.fish +++ b/share/functions/funced.fish @@ -105,8 +105,13 @@ function funced --description 'Edit function definition' echo # add a line between the parse error and the prompt set -l repeat set -l prompt (_ 'Edit the file again\? [Y/n]') - read -p "echo $prompt\ " repeat - if not contains $repeat n N no NO No nO + read -p "echo $prompt\ " response + if test -z "$response" + or contains $response {Y,y}{E,e,}{S,s,} + continue + else if not contains $response {N,n}{O,o,} + echo "I don't understand '$response', assuming 'Yes'" + sleep 2 continue end echo (_ "Cancelled function editing") From 0a23d615f4f2fe7dfb94bdf086498374ea43fc13 Mon Sep 17 00:00:00 2001 From: David Adam Date: Sat, 12 Aug 2017 12:20:49 +0800 Subject: [PATCH 20/79] docs: update language in read documentation --- doc_src/read.txt | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/doc_src/read.txt b/doc_src/read.txt index fa176c004..64a0d01c8 100644 --- a/doc_src/read.txt +++ b/doc_src/read.txt @@ -7,7 +7,10 @@ read [OPTIONS] VARIABLES... \subsection read-description Description -`read` reads from standard input and stores the result in one or more shell variables. By default it reads one line terminated by a newline but options are available to read up to a null character and to limit each "line" to a maximum number of characters. At least one variable name must be given. This command does not default to populating a var named `REPLY` like other shells do. +`read` reads from standard input and stores the result in one or more shell variables. By default, +one line (terminated by a newline) is read into each variable. Alternatively, a null character or a +maximum number of characters can be used to terminate the input. Unlike other shells, there is no +default variable (such as `REPLY`) for storing the result. The following options are available: @@ -21,7 +24,8 @@ The following options are available: - `-l` or `--local` makes the variables local. -- `-n NCHARS` or `--nchars=NCHARS` causes `read` to return after reading NCHARS characters rather than waiting for a complete line of input (either newline or null terminated). +- `-n NCHARS` or `--nchars=NCHARS` makes `read` return after reading NCHARS characters or the end of + the line, whichever comes first. - `-p PROMPT_CMD` or `--prompt=PROMPT_CMD` uses the output of the shell command `PROMPT_CMD` as the prompt for the interactive mode. The default prompt command is set_color green; echo read; set_color normal; echo "> ". @@ -39,7 +43,8 @@ The following options are available: - `-a` or `--array` stores the result as an array in a single variable. -- `-z` or `--null` reads up to NUL instead of newline. Disables interactive mode. +- `-z` or `--null` marks the end of the line with the NUL character, instead of newline. This also + disables interactive mode. `read` reads a single line of input from stdin, breaks it into tokens based on the delimiter set via `-d`/`--delimiter` as a complete string (like `string split` or, if that has not been given the (deprecated) `IFS` shell variable as a set of characters, and then assigns one token to each variable specified in `VARIABLES`. If there are more tokens than variables, the complete remainder is assigned to the last variable. As a special case, if `IFS` is set to the empty string, each character of the input is considered a separate token. @@ -47,9 +52,13 @@ If `-a` or `--array` is provided, only one variable name is allowed and the toke See the documentation for `set` for more details on the scoping rules for variables. -When read reaches the end-of-file (EOF) instead of the separator, it sets `$status` to 1. If not, it sets it to 0. +When `read` reaches the end-of-file (EOF) instead of the terminator, the exit status is set to 1. +Otherwise, it is set to 0. -Fish has a default limit of 10 MiB on the number of characters each `read` will consume. If you attempt to read more than that `$status` is set to 122 and the variable will be empty. You can modify that limit by setting the `FISH_READ_BYTE_LIMIT` variable at any time including in the environment before fish starts running. If you set it to zero then no limit is imposed. This is a safety mechanism to keep the shell from consuming an unreasonable amount of memory if the input is malformed. Note that this limit also affects how much data a command substitution is allowed to output. +In order to protect the shell from consuming too many system resources, `read` will only consume a +maximum of 10 MiB (1048576 bytes); if the terminator is not reached before this limit then VARIABLE +is set to empty and the exit status is set to 122. This limit can be altered with the +`FISH_READ_BYTE_LIMIT` variable. If set to 0 (zero), the limit is removed. \subsection read-history Using another read history file From d40667bf2601bed113125aeea79ad3471af3755c Mon Sep 17 00:00:00 2001 From: David Adam Date: Sat, 12 Aug 2017 14:26:40 +0800 Subject: [PATCH 21/79] Revert "configure: check that errno is threadsafe" This reverts commit ee15f1b9877834f49342a7bd93457e423eeb2cb4. The test relies on undefined behaviour (checking for errno in the absence of an error condition) and was broken on OpenBSD. Closes #4184. --- configure.ac | 28 ---------------------------- 1 file changed, 28 deletions(-) diff --git a/configure.ac b/configure.ac index abe5dafeb..bec1d582f 100644 --- a/configure.ac +++ b/configure.ac @@ -579,34 +579,6 @@ else AC_MSG_RESULT(no) fi -# Check that threads actually work on Solaris -AC_MSG_CHECKING([for threadsafe errno]) -AC_RUN_IFELSE( - [AC_LANG_PROGRAM([ - #include - #include - #include - - void *thread1_func(void *p_arg) - { - errno = 1; - return 0; - } - ],[ - errno = 0; - pthread_t t1; - pthread_create(&t1, NULL, thread1_func, NULL); - pthread_join(t1, NULL); - return errno; - ])], - [AC_MSG_RESULT(yes)], - [ - AC_MSG_RESULT(no) - AC_MSG_FAILURE([errno is not threadsafe - check your compiler settings]) - ], - [AC_MSG_RESULT(crosscompiling, skipped)] -) - pcre2_min_version=10.21 EXTRA_PCRE2= AC_ARG_WITH( From 47a768ceeaef1d702624802d83338edbcc0f377c Mon Sep 17 00:00:00 2001 From: Andrew Toskin Date: Sat, 12 Aug 2017 00:50:45 -0700 Subject: [PATCH 22/79] Really spell out the last of the required UNIX utilities. Finishing the job started in Pull Request #4301 Thanks to @faho for filtering and sorting my giant list of detected possible commands. --- README.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c98751418..b39b4908a 100644 --- a/README.md +++ b/README.md @@ -24,10 +24,20 @@ Compiling fish requires: At runtime, fish requires: -* a curses implementation, such as ncurses. +* a curses implementation such as ncurses (which should provide the `tput` command). * gettext, if you need the localizations. * PCRE2, due to the regular expression support contained in the `string` builtin. A copy is included with the source code, and will be used automatically if it does not already exist on your system. -* a number of common UNIX utilities: coreutils (at least ls, seq, rm, mktemp), hostname, grep, awk, sed, and getopt. +* a number of common UNIX utilities: + * an awk implementation, such as gawk, mawk, etc + * coreutils --- at least basename, cat, cut, date, dircolors, dirname, ls, mkdir, mkfifo, mktemp, rm, seq, sort, stat, stty, tail, tr, tty, uname, uniq, wc, whoami + * find + * getent (part of glibc) + * getopt + * grep + * hostname + * kill + * ps + * sed * bc, the "basic calculator" program. Optional functionality should degrade gracefully. (If fish ever threw a syntax error or a core dump due to missing dependencies for an optional feature, it would be a bug in fish and should be reported on our GitHub.) Optional features and dependencies include: From fefd1c7991c2bdd0111b8839c7c7d95aa09bc33a Mon Sep 17 00:00:00 2001 From: Mahmoud Al-Qudsi Date: Sat, 12 Aug 2017 09:54:26 -0500 Subject: [PATCH 23/79] Silence unused result warnings on newer compilers Newer versions of GCC and Clang are not satisfied by a cast to void, this fix is adapted from glibc's solution. New wrapper function ignore_result should be used when a function with explicit _unused_attribute_ wrapper is called whose result will not be handled. --- src/common.cpp | 6 +++--- src/common.h | 15 ++++++++++++++- src/env_universal_common.cpp | 4 ++-- src/path.cpp | 2 +- src/reader.cpp | 8 ++++---- src/wutil.cpp | 2 +- 6 files changed, 25 insertions(+), 12 deletions(-) diff --git a/src/common.cpp b/src/common.cpp index 2bec05227..c3a3379d7 100644 --- a/src/common.cpp +++ b/src/common.cpp @@ -616,14 +616,14 @@ void debug_safe(int level, const char *msg, const char *param1, const char *para const char *end = strchr(cursor, '%'); if (end == NULL) end = cursor + strlen(cursor); - (void)write(STDERR_FILENO, cursor, end - cursor); + ignore_result(write(STDERR_FILENO, cursor, end - cursor)); if (end[0] == '%' && end[1] == 's') { // Handle a format string. assert(param_idx < sizeof params / sizeof *params); const char *format = params[param_idx++]; if (!format) format = "(null)"; - (void)write(STDERR_FILENO, format, strlen(format)); + ignore_result(write(STDERR_FILENO, format, strlen(format))); cursor = end + 2; } else if (end[0] == '\0') { // Must be at the end of the string. @@ -635,7 +635,7 @@ void debug_safe(int level, const char *msg, const char *param1, const char *para } // We always append a newline. - (void)write(STDERR_FILENO, "\n", 1); + ignore_result(write(STDERR_FILENO, "\n", 1)); errno = errno_old; } diff --git a/src/common.h b/src/common.h index c988724ab..46953359f 100644 --- a/src/common.h +++ b/src/common.h @@ -203,7 +203,7 @@ extern bool has_working_tty_timestamps; { \ char exit_read_buff; \ show_stackframe(L'E'); \ - (void)read(0, &exit_read_buff, 1); \ + ignore_result(read(0, &exit_read_buff, 1)); \ exit_without_destructors(1); \ } @@ -876,4 +876,17 @@ enum { /// like an unrecognized flag, missing or too many arguments, an invalid integer, etc. But STATUS_INVALID_ARGS = 121, }; + +/* Normally casting an expression to void discards its value, but GCC + versions 3.4 and newer have __attribute__ ((__warn_unused_result__)) + which may cause unwanted diagnostics in that case. Use __typeof__ + and __extension__ to work around the problem, if the workaround is + known to be needed. */ +#if 3 < __GNUC__ + (4 <= __GNUC_MINOR__) +# define ignore_result(x) \ + (__extension__ ({ __typeof__ (x) __x = (x); (void) __x; })) +#else +# define ignore_result(x) ((void) (x)) +#endif + #endif diff --git a/src/env_universal_common.cpp b/src/env_universal_common.cpp index ee0bd80cb..59ca57800 100644 --- a/src/env_universal_common.cpp +++ b/src/env_universal_common.cpp @@ -1207,7 +1207,7 @@ class universal_notifier_named_pipe_t : public universal_notifier_t { // would cause us to hang! size_t read_amt = 64 * 1024; void *buff = malloc(read_amt); - (void)read(this->pipe_fd, buff, read_amt); + ignore_result(read(this->pipe_fd, buff, read_amt)); free(buff); } @@ -1306,7 +1306,7 @@ class universal_notifier_named_pipe_t : public universal_notifier_t { while (this->readback_amount > 0) { char buff[64]; size_t amt_to_read = mini(this->readback_amount, sizeof buff); - (void)read(this->pipe_fd, buff, amt_to_read); + ignore_result(read(this->pipe_fd, buff, amt_to_read)); this->readback_amount -= amt_to_read; } assert(this->readback_amount == 0); diff --git a/src/path.cpp b/src/path.cpp index f57bb0f8c..360975355 100644 --- a/src/path.cpp +++ b/src/path.cpp @@ -274,7 +274,7 @@ static void maybe_issue_path_warning(const wcstring &which_dir, const wcstring & debug(0, _(L"The error was '%s'."), strerror(saved_errno)); debug(0, _(L"Please set $%ls to a directory where you have write access."), env_var); } - (void)write(STDERR_FILENO, "\n", 1); + ignore_result(write(STDERR_FILENO, "\n", 1)); } static void path_create(wcstring &path, const wcstring &xdg_var, const wcstring &which_dir, diff --git a/src/reader.cpp b/src/reader.cpp index 7ad722e72..67daf89ff 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -696,14 +696,14 @@ void reader_write_title(const wcstring &cmd, bool reset_cursor_position) { for (size_t i = 0; i < lst.size(); i++) { fputws(lst.at(i).c_str(), stdout); } - (void)write(STDOUT_FILENO, "\a", 1); + ignore_result(write(STDOUT_FILENO, "\a", 1)); } proc_pop_interactive(); set_color(rgb_color_t::reset(), rgb_color_t::reset()); if (reset_cursor_position && !lst.empty()) { // Put the cursor back at the beginning of the line (issue #2453). - (void)write(STDOUT_FILENO, "\r", 1); + ignore_result(write(STDOUT_FILENO, "\r", 1)); } } @@ -1291,7 +1291,7 @@ static void reader_flash() { } reader_repaint(); - (void)write(STDOUT_FILENO, "\a", 1); + ignore_result(write(STDOUT_FILENO, "\a", 1)); pollint.tv_sec = 0; pollint.tv_nsec = 100 * 1000000; @@ -3236,7 +3236,7 @@ const wchar_t *reader_readline(int nchars) { reader_repaint_if_needed(); } - (void)write(STDOUT_FILENO, "\n", 1); + ignore_result(write(STDOUT_FILENO, "\n", 1)); // Ensure we have no pager contents when we exit. if (!data->pager.empty()) { diff --git a/src/wutil.cpp b/src/wutil.cpp index fd1586950..d777555bf 100644 --- a/src/wutil.cpp +++ b/src/wutil.cpp @@ -338,7 +338,7 @@ void safe_perror(const char *message) { safe_append(buff, safe_strerror(err), sizeof buff); safe_append(buff, "\n", sizeof buff); - (void)write(STDERR_FILENO, buff, strlen(buff)); + ignore_result(write(STDERR_FILENO, buff, strlen(buff))); errno = err; } From e80e9c6b068a7824a0ef4232fe281429b485e415 Mon Sep 17 00:00:00 2001 From: Andrew Toskin Date: Sat, 12 Aug 2017 22:40:48 -0700 Subject: [PATCH 24/79] getopt isn't used anymore, and getent itself is optional. getopt doesn't work very well in the BSDs, and getent has plenty of fallbacks to replace it when it's not available. --- README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/README.md b/README.md index b39b4908a..4c36b4e35 100644 --- a/README.md +++ b/README.md @@ -31,8 +31,6 @@ At runtime, fish requires: * an awk implementation, such as gawk, mawk, etc * coreutils --- at least basename, cat, cut, date, dircolors, dirname, ls, mkdir, mkfifo, mktemp, rm, seq, sort, stat, stty, tail, tr, tty, uname, uniq, wc, whoami * find - * getent (part of glibc) - * getopt * grep * hostname * kill From 59dbf64603c1bea5d401955799d1a1de849f0722 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 12:45:13 -0700 Subject: [PATCH 25/79] Revert "make `missing_var` a singleton" This reverts commit 3df8643c31a4053aa04410d967886a5888adb1e0. It was meant for the major branch. --- src/common.cpp | 1 - src/env.cpp | 33 +++++++++++---------------------- src/env.h | 36 +++++++++++++++++++----------------- src/env_universal_common.cpp | 10 ++++++---- src/expand.cpp | 4 ++-- src/fish_tests.cpp | 2 +- src/highlight.cpp | 2 +- 7 files changed, 40 insertions(+), 48 deletions(-) diff --git a/src/common.cpp b/src/common.cpp index c3a3379d7..108901929 100644 --- a/src/common.cpp +++ b/src/common.cpp @@ -1581,7 +1581,6 @@ static void validate_new_termsize(struct winsize *new_termsize) { /// Export the new terminal size as env vars and to the kernel if possible. static void export_new_termsize(struct winsize *new_termsize) { wchar_t buf[64]; - env_var_t cols = env_get(L"COLUMNS", ENV_EXPORT); swprintf(buf, 64, L"%d", (int)new_termsize->ws_col); env_set(L"COLUMNS", ENV_GLOBAL | (cols.missing_or_empty() ? 0 : ENV_EXPORT), buf); diff --git a/src/env.cpp b/src/env.cpp index 5a4b1b0d9..c5d895bf9 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -107,15 +107,6 @@ static void init_locale(); static void init_curses(); static void tokenize_variable_array(const wcstring &val, wcstring_list_t &out); -/// Construct a missing var object -env_var_t create_missing_var() { - env_var_t var; - var.set_missing(); - return var; -} - -env_var_t missing_var = create_missing_var(); - // Struct representing one level in the function variable stack. // Only our variable stack should create and destroy these class env_node_t { @@ -315,7 +306,7 @@ static bool is_electric(const wcstring &key) { const env_var_t env_node_t::find_entry(const wcstring &key) { var_table_t::const_iterator entry = env.find(key); if (entry != env.end()) return entry->second; - return missing_var; + return env_var_t::missing_var(); } /// Return the current umask value. @@ -1218,7 +1209,7 @@ int env_remove(const wcstring &key, int var_mode) { } wcstring env_var_t::as_string(void) const { - assert(!is_missing); + //assert(!is_missing); return val; } @@ -1244,12 +1235,12 @@ env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { // Make the assumption that electric keys can't be shadowed elsewhere, since we currently block // that in env_set(). if (is_electric(key)) { - if (!search_global) return missing_var; + if (!search_global) return env_var_t::missing_var(); if (key == L"history") { // Big hack. We only allow getting the history on the main thread. Note that history_t // may ask for an environment variable, so don't take the lock here (we don't need it). if (!is_main_thread()) { - return missing_var; + return env_var_t::missing_var(); } history_t *history = reader_get_history(); @@ -1289,7 +1280,7 @@ env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { } } - if (!search_universal) return missing_var; + if (!search_universal) return env_var_t::missing_var(); // Another hack. Only do a universal barrier on the main thread (since it can change variable // values). Make sure we do this outside the env_lock because it may itself call `env_get()`. @@ -1298,17 +1289,15 @@ env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { env_universal_barrier(); } - // Okay, we couldn't find a local or global var given the requirements. If there is a matching - // universal var return that. if (uvars()) { env_var_t env_var = uvars()->get(key); - if (!env_var.missing() && - (uvars()->get_export(key) ? search_exported : search_unexported)) { - return env_var; + if (env_var == ENV_NULL || + !(uvars()->get_export(key) ? search_exported : search_unexported)) { + env_var = env_var_t::missing_var(); } + return env_var; } - - return missing_var; + return env_var_t::missing_var(); } bool env_exist(const wchar_t *key, env_mode_flags_t mode) { @@ -1572,7 +1561,7 @@ env_var_t env_vars_snapshot_t::get(const wcstring &key) const { return env_get(key); } std::map::const_iterator iter = vars.find(key); - return iter == vars.end() ? missing_var : env_var_t(iter->second); + return iter == vars.end() ? env_var_t::missing_var() : env_var_t(iter->second); } const wchar_t *const env_vars_snapshot_t::highlighting_keys[] = {L"PATH", L"CDPATH", diff --git a/src/env.h b/src/env.h index e29dda483..23a602bee 100644 --- a/src/env.h +++ b/src/env.h @@ -77,13 +77,19 @@ class env_var_t { public: bool exportv; // whether the variable should be exported + static env_var_t missing_var() { + env_var_t result((wcstring())); + result.is_missing = true; + result.exportv = false; + return result; + } + + env_var_t(const env_var_t &x) : val(x.val), is_missing(x.is_missing), exportv(x.exportv) {} env_var_t(const wcstring &x) : val(x), is_missing(false), exportv(false) {} env_var_t(const wchar_t *x) : val(x), is_missing(false), exportv(false) {} env_var_t() : val(L""), is_missing(false), exportv(false) {} - void set_missing(void) { is_missing = true; } - - bool empty(void) const { return val.empty() || val == ENV_NULL; } + bool empty(void) const { return val.empty() || val == ENV_NULL; }; bool missing(void) const { return is_missing; } bool missing_or_empty(void) const { return missing() || empty(); } @@ -91,32 +97,28 @@ class env_var_t { void to_list(wcstring_list_t &out) const; wcstring as_string() const; - // You cannot convert a missing var to a non-missing var. You can only change the value of a var - // that is not missing. - void set_val(const wcstring &s) { - assert(!is_missing); - val = s; - } - void set_val(const wchar_t *s) { - assert(!is_missing); - val = s; + env_var_t &operator=(const env_var_t &v) { + is_missing = v.is_missing; + exportv = v.exportv; + val = v.val; + return *this; } + void set_val(const wcstring &s) { val = s; is_missing = false; } + void set_val(const wchar_t *s) { val = s; is_missing = false; } + bool operator==(const env_var_t &s) const { return is_missing == s.is_missing && val == s.val; } bool operator==(const wcstring &s) const { return !is_missing && val == s; } - bool operator==(const wchar_t *s) const { return !is_missing && val == s; } - bool operator!=(const env_var_t &v) const { return val != v.val; } bool operator!=(const wcstring &s) const { return val != s; } bool operator!=(const wchar_t *s) const { return val != s; } -}; -env_var_t create_missing_var(); -extern env_var_t missing_var; + bool operator==(const wchar_t *s) const { return !is_missing && val == s; } +}; /// Gets the variable with the specified name, or env_var_t::missing_var if it does not exist or is /// an empty array. diff --git a/src/env_universal_common.cpp b/src/env_universal_common.cpp index 59ca57800..8284753c3 100644 --- a/src/env_universal_common.cpp +++ b/src/env_universal_common.cpp @@ -270,9 +270,12 @@ env_universal_t::env_universal_t(const wcstring &path) env_universal_t::~env_universal_t() { pthread_mutex_destroy(&lock); } env_var_t env_universal_t::get(const wcstring &name) const { + env_var_t result = env_var_t::missing_var(); var_table_t::const_iterator where = vars.find(name); - if (where != vars.end()) return where->second; - return missing_var; + if (where != vars.end()) { + result = where->second; + } + return result; } bool env_universal_t::get_export(const wcstring &name) const { @@ -462,8 +465,7 @@ bool env_universal_t::write_to_fd(int fd, const wcstring &path) { // variable; soldier on. const wcstring &key = iter->first; const env_var_t &var = iter->second; - append_file_entry(var.exportv ? SET_EXPORT : SET, key, var.as_string(), &contents, - &storage); + append_file_entry(var.exportv ? SET_EXPORT : SET, key, var.as_string(), &contents, &storage); // Go to next. ++iter; diff --git a/src/expand.cpp b/src/expand.cpp index 9de0a38d7..2bcc6ec18 100644 --- a/src/expand.cpp +++ b/src/expand.cpp @@ -144,7 +144,7 @@ static void append_cmdsub_error(parse_error_list_t *errors, size_t source_start, /// Return the environment variable value for the string starting at \c in. static env_var_t expand_var(const wchar_t *in) { - if (!in) return missing_var; + if (!in) return env_var_t::missing_var(); return env_get(in); } @@ -777,7 +777,7 @@ static int expand_variables(const wcstring &instr, std::vector *ou var_tmp.append(instr, start_pos, var_len); env_var_t var; if (var_len == 1 && var_tmp[0] == VARIABLE_EXPAND_EMPTY) { - var = missing_var; + var = env_var_t::missing_var(); } else { var = expand_var(var_tmp.c_str()); } diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index c7e6c200f..822d71b99 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -2658,7 +2658,7 @@ static void test_universal() { const wcstring key = format_string(L"key_%d_%d", i, j); env_var_t expected_val; if (j == 0) { - expected_val = missing_var; + expected_val = env_var_t::missing_var(); } else { expected_val = format_string(L"val_%d_%d", i, j); } diff --git a/src/highlight.cpp b/src/highlight.cpp index 7186db127..aefafa436 100644 --- a/src/highlight.cpp +++ b/src/highlight.cpp @@ -222,7 +222,7 @@ static bool is_potential_cd_path(const wcstring &path, const wcstring &working_d } else { // Get the CDPATH. env_var_t cdpath = env_get(L"CDPATH"); - if (cdpath.missing_or_empty()) cdpath = env_var_t(L"."); + if (cdpath.missing_or_empty()) cdpath = L"."; // Tokenize it into directories. std::vector pathsv; From 9f4f9545c1cbd63bf8805fc40dc7f43f4a4f9b7b Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 12:45:49 -0700 Subject: [PATCH 26/79] Revert "change order of `env_set()` args" This reverts commit 6e7956a41383b3273981916901c6db091ea0c6ed. It was meant for the major branch. --- src/builtin_argparse.cpp | 14 ++++++------- src/builtin_fg.cpp | 2 +- src/builtin_read.cpp | 22 ++++++++++---------- src/builtin_set.cpp | 10 ++++----- src/common.cpp | 4 ++-- src/env.cpp | 44 ++++++++++++++++++++-------------------- src/env.h | 2 +- src/exec.cpp | 2 +- src/fish.cpp | 2 +- src/fish_tests.cpp | 6 +++--- src/function.cpp | 4 ++-- src/input.cpp | 2 +- src/parse_execution.cpp | 2 +- src/path.cpp | 2 +- src/reader.cpp | 12 +++++------ 15 files changed, 65 insertions(+), 65 deletions(-) diff --git a/src/builtin_argparse.cpp b/src/builtin_argparse.cpp index f1d934b65..55e7afc4d 100644 --- a/src/builtin_argparse.cpp +++ b/src/builtin_argparse.cpp @@ -458,13 +458,13 @@ static int validate_arg(argparse_cmd_opts_t &opts, option_spec_t *opt_spec, bool wcstring_list_t cmd_output; env_push(true); - env_set(L"_argparse_cmd", ENV_LOCAL, opts.name.c_str()); + env_set(L"_argparse_cmd", opts.name.c_str(), ENV_LOCAL); if (is_long_flag) { - env_set(var_name_prefix + L"name", ENV_LOCAL, opt_spec->long_flag.c_str()); + env_set(var_name_prefix + L"name", opt_spec->long_flag.c_str(), ENV_LOCAL); } else { - env_set(var_name_prefix + L"name", ENV_LOCAL, wcstring(1, opt_spec->short_flag).c_str()); + env_set(var_name_prefix + L"name", wcstring(1, opt_spec->short_flag).c_str(), ENV_LOCAL); } - env_set(var_name_prefix + L"value", ENV_LOCAL, woptarg); + env_set(var_name_prefix + L"value", woptarg, ENV_LOCAL); int retval = exec_subshell(opt_spec->validation_command, cmd_output, false); for (auto it : cmd_output) { @@ -645,7 +645,7 @@ static void set_argparse_result_vars(argparse_cmd_opts_t &opts) { auto val = list_to_array_val(opt_spec->vals); if (opt_spec->short_flag_valid) { - env_set(var_name_prefix + opt_spec->short_flag, ENV_LOCAL, val->c_str()); + env_set(var_name_prefix + opt_spec->short_flag, val->c_str(), ENV_LOCAL); } if (!opt_spec->long_flag.empty()) { // We do a simple replacement of all non alphanum chars rather than calling @@ -654,12 +654,12 @@ static void set_argparse_result_vars(argparse_cmd_opts_t &opts) { for (size_t pos = 0; pos < long_flag.size(); pos++) { if (!iswalnum(long_flag[pos])) long_flag[pos] = L'_'; } - env_set(var_name_prefix + long_flag, ENV_LOCAL, val->c_str()); + env_set(var_name_prefix + long_flag, val->c_str(), ENV_LOCAL); } } auto val = list_to_array_val(opts.argv); - env_set(L"argv", ENV_LOCAL, val->c_str()); + env_set(L"argv", val->c_str(), ENV_LOCAL); } /// The argparse builtin. This is explicitly not compatible with the BSD or GNU version of this diff --git a/src/builtin_fg.cpp b/src/builtin_fg.cpp index 33255cf71..4fb05c846 100644 --- a/src/builtin_fg.cpp +++ b/src/builtin_fg.cpp @@ -102,7 +102,7 @@ int builtin_fg(parser_t &parser, io_streams_t &streams, wchar_t **argv) { } const wcstring ft = tok_first(j->command()); - if (!ft.empty()) env_set(L"_", ENV_EXPORT, ft.c_str()); + if (!ft.empty()) env_set(L"_", ft.c_str(), ENV_EXPORT); reader_write_title(j->command()); job_promote(j); diff --git a/src/builtin_read.cpp b/src/builtin_read.cpp index 6e50e3a7f..7d6a0522a 100644 --- a/src/builtin_read.cpp +++ b/src/builtin_read.cpp @@ -422,7 +422,7 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { if (exit_res != STATUS_CMD_OK) { // Define the var(s) without any data. We do this because when this happens we want the user // to be able to use the var but have it expand to nothing. - for (int i = 0; i < argc; i++) env_set(argv[i], opts.place, NULL); + for (int i = 0; i < argc; i++) env_set(argv[i], NULL, opts.place); return exit_res; } @@ -444,9 +444,9 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { *out = *it; out += 2; } - env_set(argv[0], opts.place, chars.c_str()); + env_set(argv[0], chars.c_str(), opts.place); } else { - env_set(argv[0], opts.place, NULL); + env_set(argv[0], NULL, opts.place); } } else { // not array mode int i = 0; @@ -454,12 +454,12 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { for (; i + 1 < argc; ++i) { if (j < bufflen) { wchar_t buffer[2] = {buff[j++], 0}; - env_set(argv[i], opts.place, buffer); + env_set(argv[i], buffer, opts.place); } else { - env_set(argv[i], opts.place, L""); + env_set(argv[i], L"", opts.place); } } - if (i < argc) env_set(argv[i], opts.place, &buff[j]); + if (i < argc) env_set(argv[i], &buff[j], opts.place); } } else if (opts.array) { if (!opts.have_delimiter) { @@ -473,21 +473,21 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { if (!tokens.empty()) tokens.push_back(ARRAY_SEP); tokens.append(buff, loc.first, loc.second); } - env_set(argv[0], opts.place, tokens.empty() ? NULL : tokens.c_str()); + env_set(argv[0], tokens.empty() ? NULL : tokens.c_str(), opts.place); } else { wcstring_list_t splits; split_about(buff.begin(), buff.end(), opts.delimiter.begin(), opts.delimiter.end(), &splits, LONG_MAX); auto val = list_to_array_val(splits); - env_set(argv[0], opts.place, val->c_str()); + env_set(argv[0], val->c_str(), opts.place); } } else { // not array if (!opts.have_delimiter) { wcstring_range loc = wcstring_range(0, 0); for (int i = 0; i < argc; i++) { loc = wcstring_tok(buff, (i + 1 < argc) ? opts.delimiter : wcstring(), loc); - env_set(argv[i], opts.place, - loc.first == wcstring::npos ? L"" : &buff.c_str()[loc.first]); + env_set(argv[i], loc.first == wcstring::npos ? L"" : &buff.c_str()[loc.first], + opts.place); } } else { wcstring_list_t splits; @@ -496,7 +496,7 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { split_about(buff.begin(), buff.end(), opts.delimiter.begin(), opts.delimiter.end(), &splits, argc - 1); for (size_t i = 0; i < (size_t)argc && i < splits.size(); i++) { - env_set(argv[i], opts.place, splits[i].c_str()); + env_set(argv[i], splits[i].c_str(), opts.place); } } } diff --git a/src/builtin_set.cpp b/src/builtin_set.cpp index 41ab75aeb..8672e6fcc 100644 --- a/src/builtin_set.cpp +++ b/src/builtin_set.cpp @@ -284,8 +284,8 @@ static int my_env_path_setup(const wchar_t *cmd, const wchar_t *key, //!OCLINT( /// Call env_set. If this is a path variable, e.g. PATH, validate the elements. On error, print a /// description of the problem to stderr. -static int my_env_set(const wchar_t *cmd, const wchar_t *key, int scope, - const wcstring_list_t &list, io_streams_t &streams) { +static int my_env_set(const wchar_t *cmd, const wchar_t *key, const wcstring_list_t &list, + int scope, io_streams_t &streams) { int retval; if (is_path_variable(key)) { @@ -294,7 +294,7 @@ static int my_env_set(const wchar_t *cmd, const wchar_t *key, int scope, } auto val = list_to_array_val(list); - retval = env_set(key, scope | ENV_USER, val->c_str()); + retval = env_set(key, val->c_str(), scope | ENV_USER); switch (retval) { case ENV_OK: { retval = STATUS_CMD_OK; @@ -633,7 +633,7 @@ static int builtin_set_erase(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, wcstring_list_t result; dest_var.to_list(result); erase_values(result, indexes); - retval = my_env_set(cmd, dest, scope, result, streams); + retval = my_env_set(cmd, dest, result, scope, streams); } if (retval != STATUS_CMD_OK) return retval; @@ -742,7 +742,7 @@ static int builtin_set_set(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, w } if (retval != STATUS_CMD_OK) return retval; - retval = my_env_set(cmd, varname, scope, new_values, streams); + retval = my_env_set(cmd, varname, new_values, scope, streams); if (retval != STATUS_CMD_OK) return retval; return check_global_scope_exists(cmd, opts, varname, streams); } diff --git a/src/common.cpp b/src/common.cpp index 108901929..213bdf479 100644 --- a/src/common.cpp +++ b/src/common.cpp @@ -1583,11 +1583,11 @@ static void export_new_termsize(struct winsize *new_termsize) { wchar_t buf[64]; env_var_t cols = env_get(L"COLUMNS", ENV_EXPORT); swprintf(buf, 64, L"%d", (int)new_termsize->ws_col); - env_set(L"COLUMNS", ENV_GLOBAL | (cols.missing_or_empty() ? 0 : ENV_EXPORT), buf); + env_set(L"COLUMNS", buf, ENV_GLOBAL | (cols.missing_or_empty() ? 0 : ENV_EXPORT)); env_var_t lines = env_get(L"LINES", ENV_EXPORT); swprintf(buf, 64, L"%d", (int)new_termsize->ws_row); - env_set(L"LINES", ENV_GLOBAL | (lines.missing_or_empty() ? 0 : ENV_EXPORT), buf); + env_set(L"LINES", buf, ENV_GLOBAL | (lines.missing_or_empty() ? 0 : ENV_EXPORT)); #ifdef HAVE_WINSIZE ioctl(STDOUT_FILENO, TIOCSWINSZ, new_termsize); diff --git a/src/env.cpp b/src/env.cpp index c5d895bf9..1fa680d6b 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -369,7 +369,7 @@ static void fix_colon_delimited_var(const wcstring &var_name) { } } - int scope = env_set(var_name, ENV_USER, new_val.c_str()); + int scope = env_set(var_name, new_val.c_str(), ENV_USER); if (scope != ENV_OK) { debug(0, L"fix_colon_delimited_var failed unexpectedly with retval %d", scope); } @@ -667,7 +667,7 @@ static void setup_path() { const env_var_t path = env_get(L"PATH"); if (path.missing_or_empty()) { const wchar_t *value = L"/usr/bin" ARRAY_SEP_STR L"/bin"; - env_set(L"PATH", ENV_GLOBAL | ENV_EXPORT, value); + env_set(L"PATH", value, ENV_GLOBAL | ENV_EXPORT); } } @@ -676,10 +676,10 @@ static void setup_path() { /// adjusted. static void env_set_termsize() { env_var_t cols = env_get(L"COLUMNS"); - if (cols.missing_or_empty()) env_set(L"COLUMNS", ENV_GLOBAL, DFLT_TERM_COL_STR); + if (cols.missing_or_empty()) env_set(L"COLUMNS", DFLT_TERM_COL_STR, ENV_GLOBAL); env_var_t rows = env_get(L"LINES"); - if (rows.missing_or_empty()) env_set(L"LINES", ENV_GLOBAL, DFLT_TERM_ROW_STR); + if (rows.missing_or_empty()) env_set(L"LINES", DFLT_TERM_ROW_STR, ENV_GLOBAL); } bool env_set_pwd() { @@ -689,7 +689,7 @@ bool env_set_pwd() { _(L"Could not determine current working directory. Is your locale set correctly?")); return false; } - env_set(L"PWD", ENV_EXPORT | ENV_GLOBAL, res.c_str()); + env_set(L"PWD", res.c_str(), ENV_EXPORT | ENV_GLOBAL); return true; } @@ -728,7 +728,7 @@ static void setup_user(bool force) { int retval = getpwuid_r(getuid(), &userinfo, buf, sizeof(buf), &result); if (!retval && result) { const wcstring uname = str2wcstring(userinfo.pw_name); - env_set(L"USER", ENV_GLOBAL | ENV_EXPORT, uname.c_str()); + env_set(L"USER", uname.c_str(), ENV_GLOBAL | ENV_EXPORT); } } } @@ -820,7 +820,7 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { if (eql == wcstring::npos) { // No equals found. if (is_read_only(key_and_val) || is_electric(key_and_val)) continue; - env_set(key_and_val, ENV_EXPORT | ENV_GLOBAL, L""); + env_set(key_and_val, L"", ENV_EXPORT | ENV_GLOBAL); } else { key.assign(key_and_val, 0, eql); if (is_read_only(key) || is_electric(key)) continue; @@ -829,16 +829,16 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { std::replace(val.begin(), val.end(), L':', ARRAY_SEP); } - env_set(key, ENV_EXPORT | ENV_GLOBAL, val.c_str()); + env_set(key, val.c_str(), ENV_EXPORT | ENV_GLOBAL); } } // Set the given paths in the environment, if we have any. if (paths != NULL) { - env_set(FISH_DATADIR_VAR, ENV_GLOBAL, paths->data.c_str()); - env_set(FISH_SYSCONFDIR_VAR, ENV_GLOBAL, paths->sysconf.c_str()); - env_set(FISH_HELPDIR_VAR, ENV_GLOBAL, paths->doc.c_str()); - env_set(FISH_BIN_DIR, ENV_GLOBAL, paths->bin.c_str()); + env_set(FISH_DATADIR_VAR, paths->data.c_str(), ENV_GLOBAL); + env_set(FISH_SYSCONFDIR_VAR, paths->sysconf.c_str(), ENV_GLOBAL); + env_set(FISH_HELPDIR_VAR, paths->doc.c_str(), ENV_GLOBAL); + env_set(FISH_BIN_DIR, paths->bin.c_str(), ENV_GLOBAL); } init_locale(); @@ -859,7 +859,7 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { // Set up the version variable. wcstring version = str2wcstring(get_fish_version()); - env_set(L"FISH_VERSION", ENV_GLOBAL, version.c_str()); + env_set(L"FISH_VERSION", version.c_str(), ENV_GLOBAL); // Set up SHLVL variable. const env_var_t shlvl_var = env_get(L"SHLVL"); @@ -872,7 +872,7 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { nshlvl_str = to_string(shlvl_i + 1); } } - env_set(L"SHLVL", ENV_GLOBAL | ENV_EXPORT, nshlvl_str.c_str()); + env_set(L"SHLVL", nshlvl_str.c_str(), ENV_GLOBAL | ENV_EXPORT); env_read_only.insert(L"SHLVL"); // Set up the HOME variable. @@ -899,18 +899,18 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { } if (!retval && result && userinfo.pw_dir) { const wcstring dir = str2wcstring(userinfo.pw_dir); - env_set(L"HOME", ENV_GLOBAL | ENV_EXPORT, dir.c_str()); + env_set(L"HOME", dir.c_str(), ENV_GLOBAL | ENV_EXPORT); } else { // We cannot get $HOME, set it to the empty list. // This triggers warnings for history and config.fish already, // so it isn't necessary to warn here as well. - env_set(L"HOME", ENV_GLOBAL | ENV_EXPORT, ENV_NULL); + env_set(L"HOME", ENV_NULL, ENV_GLOBAL | ENV_EXPORT); } free(unam_narrow); } else { // If $USER is empty as well (which we tried to set above), // we can't get $HOME. - env_set(L"HOME", ENV_GLOBAL | ENV_EXPORT, ENV_NULL); + env_set(L"HOME", ENV_NULL, ENV_GLOBAL | ENV_EXPORT); } } @@ -924,7 +924,7 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { use_posix_spawn.missing_or_empty() ? true : from_string(use_posix_spawn.as_string()); // Set fish_bind_mode to "default". - env_set(FISH_BIND_MODE_VAR, ENV_GLOBAL, DEFAULT_BIND_MODE); + env_set(FISH_BIND_MODE_VAR, DEFAULT_BIND_MODE, ENV_GLOBAL); // This is somewhat subtle. At this point we consider our environment to be sufficiently // initialized that we can react to changes to variables. Prior to doing this we expect that the @@ -976,7 +976,7 @@ static env_node_t *env_get_node(const wcstring &key) { /// * ENV_SCOPE, the variable cannot be set in the given scope. This applies to readonly/electric /// variables set from the local or universal scopes, or set as exported. /// * ENV_INVALID, the variable value was invalid. This applies only to special variables. -int env_set(const wcstring &key, env_mode_flags_t var_mode, const wchar_t *val) { +int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t var_mode) { ASSERT_IS_MAIN_THREAD(); bool has_changed_old = vars_stack().has_changed_exported; int done = 0; @@ -986,7 +986,7 @@ int env_set(const wcstring &key, env_mode_flags_t var_mode, const wchar_t *val) wcstring val_canonical = val; path_make_canonical(val_canonical); if (val != val_canonical) { - return env_set(key, var_mode, val_canonical.c_str()); + return env_set(key, val_canonical.c_str(), var_mode); } } @@ -1528,9 +1528,9 @@ void env_set_argv(const wchar_t *const *argv) { sb.append(*arg); } - env_set(L"argv", ENV_LOCAL, sb.c_str()); + env_set(L"argv", sb.c_str(), ENV_LOCAL); } else { - env_set(L"argv", ENV_LOCAL, NULL); + env_set(L"argv", NULL, ENV_LOCAL); } } diff --git a/src/env.h b/src/env.h index 23a602bee..5c5890991 100644 --- a/src/env.h +++ b/src/env.h @@ -127,7 +127,7 @@ class env_var_t { /// \param mode An optional scope to search in. All scopes are searched if unset env_var_t env_get(const wcstring &key, env_mode_flags_t mode = ENV_DEFAULT); -int env_set(const wcstring &key, env_mode_flags_t mode, const wchar_t *val); +int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t mode); /// Returns true if the specified key exists. This can't be reliably done using env_get, since /// env_get returns null for 0-element arrays. diff --git a/src/exec.cpp b/src/exec.cpp index aa82f1bf0..dc452d3b3 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -384,7 +384,7 @@ void internal_exec(job_t *j, const io_chain_t &&all_ios) { shlvl_str = to_string(shlvl - 1); } } - env_set(L"SHLVL", ENV_GLOBAL | ENV_EXPORT, shlvl_str.c_str()); + env_set(L"SHLVL", shlvl_str.c_str(), ENV_GLOBAL | ENV_EXPORT); // launch_process _never_ returns. launch_process_nofork(j->processes.front().get()); diff --git a/src/fish.cpp b/src/fish.cpp index 1a45cb9f4..a3f45dfe3 100644 --- a/src/fish.cpp +++ b/src/fish.cpp @@ -421,7 +421,7 @@ int main(int argc, char **argv) { list.push_back(str2wcstring(*ptr)); } auto val = list_to_array_val(list); - env_set(L"argv", ENV_DEFAULT, val->c_str()); + env_set(L"argv", val->c_str(), ENV_DEFAULT); const wcstring rel_filename = str2wcstring(file); diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 822d71b99..9c539d96e 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -1617,7 +1617,7 @@ static void test_abbreviations(void) { {L"gc", L"git checkout"}, {L"foo", L"bar"}, {L"gx", L"git checkout"}, }; for (auto it : abbreviations) { - int ret = env_set(L"_fish_abbr_" + it.first, ENV_LOCAL, it.second.c_str()); + int ret = env_set(L"_fish_abbr_" + it.first, it.second.c_str(), ENV_LOCAL); if (ret != 0) err(L"Unable to set abbreviation variable"); } @@ -2479,7 +2479,7 @@ static void test_autosuggest_suggest_special() { perform_one_autosuggestion_cd_test(L"cd 'test/autosuggest_test/5", vars, L"foo\"bar/", __LINE__); - env_set(L"AUTOSUGGEST_TEST_LOC", ENV_LOCAL, wd.c_str()); + env_set(L"AUTOSUGGEST_TEST_LOC", wd.c_str(), ENV_LOCAL); perform_one_autosuggestion_cd_test(L"cd $AUTOSUGGEST_TEST_LOC/0", vars, L"foobar/", __LINE__); perform_one_autosuggestion_cd_test(L"cd ~/test_autosuggest_suggest_specia", vars, L"l/", __LINE__); @@ -4255,7 +4255,7 @@ long return_timezone_hour(time_t tstamp, const wchar_t *timezone) { char *str_ptr; size_t n; - env_set(L"TZ", ENV_EXPORT, timezone); + env_set(L"TZ", timezone, ENV_EXPORT); localtime_r(&tstamp, <ime); n = strftime(ltime_str, 3, "%H", <ime); if (n != 2) { diff --git a/src/function.cpp b/src/function.cpp index 9dd3d8264..44105c130 100644 --- a/src/function.cpp +++ b/src/function.cpp @@ -340,7 +340,7 @@ void function_prepare_environment(const wcstring &name, const wchar_t *const *ar const wchar_t *const *arg; size_t i; for (i = 0, arg = argv; i < named_arguments.size(); i++) { - env_set(named_arguments.at(i).c_str(), ENV_LOCAL | ENV_USER, *arg); + env_set(named_arguments.at(i).c_str(), *arg, ENV_LOCAL | ENV_USER); if (*arg) arg++; } @@ -349,6 +349,6 @@ void function_prepare_environment(const wcstring &name, const wchar_t *const *ar for (std::map::const_iterator it = inherited_vars.begin(), end = inherited_vars.end(); it != end; ++it) { - env_set(it->first, ENV_LOCAL | ENV_USER, it->second.missing() ? NULL : it->second.c_str()); + env_set(it->first, it->second.missing() ? NULL : it->second.c_str(), ENV_LOCAL | ENV_USER); } } diff --git a/src/input.cpp b/src/input.cpp index 456e07766..94d4a4469 100644 --- a/src/input.cpp +++ b/src/input.cpp @@ -211,7 +211,7 @@ void input_set_bind_mode(const wcstring &bm) { // modes may not be empty - empty is a sentinel value meaning to not change the mode assert(!bm.empty()); if (input_get_bind_mode() != bm.c_str()) { - env_set(FISH_BIND_MODE_VAR, ENV_GLOBAL, bm.c_str()); + env_set(FISH_BIND_MODE_VAR, bm.c_str(), ENV_GLOBAL); } } diff --git a/src/parse_execution.cpp b/src/parse_execution.cpp index 1b1557711..f21327569 100644 --- a/src/parse_execution.cpp +++ b/src/parse_execution.cpp @@ -473,7 +473,7 @@ parse_execution_result_t parse_execution_context_t::run_for_statement( } const wcstring &val = argument_sequence.at(i); - env_set(for_var_name, ENV_LOCAL, val.c_str()); + env_set(for_var_name, val.c_str(), ENV_LOCAL); fb->loop_status = LOOP_NORMAL; this->run_job_list(block_contents, fb); diff --git a/src/path.cpp b/src/path.cpp index 360975355..70391067d 100644 --- a/src/path.cpp +++ b/src/path.cpp @@ -260,7 +260,7 @@ static void maybe_issue_path_warning(const wcstring &which_dir, const wcstring & if (env_exist(warning_var_name.c_str(), ENV_GLOBAL | ENV_EXPORT)) { return; } - env_set(warning_var_name, ENV_GLOBAL | ENV_EXPORT, L"1"); + env_set(warning_var_name, L"1", ENV_GLOBAL | ENV_EXPORT); debug(0, custom_error_msg.c_str()); if (path.empty()) { diff --git a/src/reader.cpp b/src/reader.cpp index 67daf89ff..7daccce2a 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -765,8 +765,8 @@ void reader_init() { // Ensure this var is present even before an interactive command is run so that if it is used // in a function like `fish_prompt` or `fish_right_prompt` it is defined at the time the first - // prompt is written. - env_set(ENV_CMD_DURATION, ENV_UNEXPORT, L"0"); + // prompt is issued. + env_set(ENV_CMD_DURATION, L"0", ENV_UNEXPORT); // Save the initial terminal mode. tcgetattr(STDIN_FILENO, &terminal_mode_on_startup); @@ -1650,7 +1650,7 @@ static void reader_interactive_init() { wperror(L"tcsetattr"); } - env_set(L"_", ENV_GLOBAL, L"fish"); + env_set(L"_", L"fish", ENV_GLOBAL); } /// Destroy data for interactive use. @@ -1917,7 +1917,7 @@ void set_env_cmd_duration(struct timeval *after, struct timeval *before) { } swprintf(buf, 16, L"%d", (secs * 1000) + (usecs / 1000)); - env_set(ENV_CMD_DURATION, ENV_UNEXPORT, buf); + env_set(ENV_CMD_DURATION, buf, ENV_UNEXPORT); } void reader_run_command(parser_t &parser, const wcstring &cmd) { @@ -1925,7 +1925,7 @@ void reader_run_command(parser_t &parser, const wcstring &cmd) { wcstring ft = tok_first(cmd); - if (!ft.empty()) env_set(L"_", ENV_GLOBAL, ft.c_str()); + if (!ft.empty()) env_set(L"_", ft.c_str(), ENV_GLOBAL); reader_write_title(cmd); @@ -1941,7 +1941,7 @@ void reader_run_command(parser_t &parser, const wcstring &cmd) { term_steal(); - env_set(L"_", ENV_GLOBAL, program_name); + env_set(L"_", program_name, ENV_GLOBAL); #ifdef HAVE__PROC_SELF_STAT proc_update_jiffies(); From a2f507f1c855cdcf5d15435162060f2b45bbdc42 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 14:57:31 -0700 Subject: [PATCH 27/79] Revert "remove more ENV_NULL references" This reverts commit 591449aba72f5f6a8fa646f57704ece4428eccb4. It was meant for the major branch. --- src/builtin_set.cpp | 2 ++ src/expand.cpp | 10 +++++++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/src/builtin_set.cpp b/src/builtin_set.cpp index 8672e6fcc..1d1af95a6 100644 --- a/src/builtin_set.cpp +++ b/src/builtin_set.cpp @@ -293,6 +293,8 @@ static int my_env_set(const wchar_t *cmd, const wchar_t *key, const wcstring_lis if (retval != STATUS_CMD_OK) return retval; } + // We don't check `val->empty()` because an array var with a single empty string will be + // "empty". A truly empty array var is set to the special value `ENV_NULL`. auto val = list_to_array_val(list); retval = env_set(key, val->c_str(), scope | ENV_USER); switch (retval) { diff --git a/src/expand.cpp b/src/expand.cpp index 2bcc6ec18..4a34ba4a2 100644 --- a/src/expand.cpp +++ b/src/expand.cpp @@ -1172,8 +1172,16 @@ static void expand_home_directory(wcstring &input) { if (username.empty()) { // Current users home directory. home = env_get(L"HOME"); + // If home is either missing or empty, + // treat it like an empty list. + // $HOME is defined to be a _path_, + // and those cannot be empty. + // + // We do not expand a string-empty var differently, + // because that results in bogus paths + // - ~/foo turns into /foo. if (home.missing_or_empty()) { - input.clear(); + input = ENV_NULL; return; } tail_idx = 1; From 2b7a4143ecf80ddf3c77fd23b773f8f893d4b754 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 14:57:44 -0700 Subject: [PATCH 28/79] Revert "fix bug in `env_get()` involving empty vars" This reverts commit 559b05d01df07f2d60caaa686d4dd1357ab231bb. It was meant for the major branch. --- Makefile.in | 4 ++-- src/builtin_argparse.cpp | 7 ++++--- src/builtin_read.cpp | 2 +- src/builtin_set.cpp | 8 +++++--- src/common.cpp | 1 + src/env.cpp | 6 ++++++ src/env.h | 4 ++-- src/expand.cpp | 20 +++++++++++--------- src/expand.h | 2 +- src/fish.cpp | 2 +- 10 files changed, 34 insertions(+), 22 deletions(-) diff --git a/Makefile.in b/Makefile.in index 513eaf33f..0ecb86acc 100644 --- a/Makefile.in +++ b/Makefile.in @@ -353,7 +353,7 @@ test: test-prep install-force test_low_level test_high_level # We want the various tests to run serially so their output doesn't mix # We can do that by adding ordering dependencies based on what goals are being used. # -test_goals := test_low_level test_fishscript test_interactive test_invocation +test_goals := test_low_level test_invocation test_fishscript test_interactive # # The following variables define targets that depend on the tests. If any more targets @@ -374,7 +374,7 @@ test_low_level: fish_tests $(call filter_up_to,test_low_level,$(active_test_goal test_high_level: DESTDIR = $(PWD)/test/root/ test_high_level: prefix = . -test_high_level: test-prep install-force test_fishscript test_interactive test_invocation +test_high_level: test-prep install-force test_invocation test_fishscript test_interactive .PHONY: test_high_level test_invocation: $(call filter_up_to,test_invocation,$(active_test_goals)) diff --git a/src/builtin_argparse.cpp b/src/builtin_argparse.cpp index 55e7afc4d..45475219d 100644 --- a/src/builtin_argparse.cpp +++ b/src/builtin_argparse.cpp @@ -645,7 +645,8 @@ static void set_argparse_result_vars(argparse_cmd_opts_t &opts) { auto val = list_to_array_val(opt_spec->vals); if (opt_spec->short_flag_valid) { - env_set(var_name_prefix + opt_spec->short_flag, val->c_str(), ENV_LOCAL); + env_set(var_name_prefix + opt_spec->short_flag, *val == ENV_NULL ? NULL : val->c_str(), + ENV_LOCAL); } if (!opt_spec->long_flag.empty()) { // We do a simple replacement of all non alphanum chars rather than calling @@ -654,12 +655,12 @@ static void set_argparse_result_vars(argparse_cmd_opts_t &opts) { for (size_t pos = 0; pos < long_flag.size(); pos++) { if (!iswalnum(long_flag[pos])) long_flag[pos] = L'_'; } - env_set(var_name_prefix + long_flag, val->c_str(), ENV_LOCAL); + env_set(var_name_prefix + long_flag, *val == ENV_NULL ? NULL : val->c_str(), ENV_LOCAL); } } auto val = list_to_array_val(opts.argv); - env_set(L"argv", val->c_str(), ENV_LOCAL); + env_set(L"argv", *val == ENV_NULL ? NULL : val->c_str(), ENV_LOCAL); } /// The argparse builtin. This is explicitly not compatible with the BSD or GNU version of this diff --git a/src/builtin_read.cpp b/src/builtin_read.cpp index 7d6a0522a..cfd32ace2 100644 --- a/src/builtin_read.cpp +++ b/src/builtin_read.cpp @@ -479,7 +479,7 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { split_about(buff.begin(), buff.end(), opts.delimiter.begin(), opts.delimiter.end(), &splits, LONG_MAX); auto val = list_to_array_val(splits); - env_set(argv[0], val->c_str(), opts.place); + env_set(argv[0], *val == ENV_NULL ? NULL : val->c_str(), opts.place); } } else { // not array if (!opts.have_delimiter) { diff --git a/src/builtin_set.cpp b/src/builtin_set.cpp index 1d1af95a6..810451033 100644 --- a/src/builtin_set.cpp +++ b/src/builtin_set.cpp @@ -457,15 +457,17 @@ static int builtin_set_list(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, if (!names_only) { env_var_t var = env_get(key, compute_scope(opts)); - if (!var.missing_or_empty()) { + if (!var.missing()) { bool shorten = false; - wcstring val = expand_escape_variable(var); + wcstring val = var.as_string(); if (opts.shorten_ok && val.length() > 64) { shorten = true; val.resize(60); } + + wcstring e_value = expand_escape_variable(val); streams.out.append(L" "); - streams.out.append(val); + streams.out.append(e_value); if (shorten) streams.out.push_back(ellipsis_char); } diff --git a/src/common.cpp b/src/common.cpp index 213bdf479..1dfbdda86 100644 --- a/src/common.cpp +++ b/src/common.cpp @@ -957,6 +957,7 @@ static void escape_string_script(const wchar_t *orig_in, size_t in_len, wcstring case L'\\': case L'\'': { need_escape = need_complex_escape = 1; + // WTF if (escape_all) out += L'\\'; out += L'\\'; out += *in; break; diff --git a/src/env.cpp b/src/env.cpp index 1fa680d6b..63389d4ed 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -1268,6 +1268,12 @@ env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { while (env != NULL) { const env_var_t var = env->find_entry(key); if (!var.missing() && (var.exportv ? search_exported : search_unexported)) { + // The following statement is wrong because ENV_NULL is supposed to mean the var is + // set but has zero elements. Therefore we should not return missing_var. However, + // If this is changed to return `var` without the special-case then unit tests fail. + // Note that tokenize_variable_array() explicitly handles a var whose string + // representation contains only ENV_NULL. + if (var.as_string() == ENV_NULL) return env_var_t::missing_var(); return var; } diff --git a/src/env.h b/src/env.h index 5c5890991..aee366f6c 100644 --- a/src/env.h +++ b/src/env.h @@ -89,9 +89,9 @@ class env_var_t { env_var_t(const wchar_t *x) : val(x), is_missing(false), exportv(false) {} env_var_t() : val(L""), is_missing(false), exportv(false) {} - bool empty(void) const { return val.empty() || val == ENV_NULL; }; + bool empty(void) const { return val.empty(); }; bool missing(void) const { return is_missing; } - bool missing_or_empty(void) const { return missing() || empty(); } + bool missing_or_empty(void) const { return missing() || val.empty(); } const wchar_t *c_str(void) const; void to_list(wcstring_list_t &out) const; diff --git a/src/expand.cpp b/src/expand.cpp index 4a34ba4a2..f265485d4 100644 --- a/src/expand.cpp +++ b/src/expand.cpp @@ -27,10 +27,8 @@ #include #include -#include #include // IWYU pragma: keep #include -#include #include #include "common.h" @@ -44,6 +42,7 @@ #include "parse_util.h" #include "path.h" #include "proc.h" +#include "util.h" #include "wildcard.h" #include "wutil.h" // IWYU pragma: keep #ifdef KERN_PROCARGS2 @@ -170,13 +169,15 @@ static int is_quotable(const wchar_t *str) { static int is_quotable(const wcstring &str) { return is_quotable(str.c_str()); } wcstring expand_escape_variable(const env_var_t &var) { - wcstring buff; wcstring_list_t lst; + wcstring buff; var.to_list(lst); - if (lst.size() == 0) { - ; // empty list expands to nothing - } else if (lst.size() == 1) { + + size_t size = lst.size(); + if (size == 0) { + buff.append(L"''"); + } else if (size == 1) { const wcstring &el = lst.at(0); if (el.find(L' ') != wcstring::npos && is_quotable(el)) { @@ -200,7 +201,6 @@ wcstring expand_escape_variable(const env_var_t &var) { } } } - return buff; } @@ -265,7 +265,9 @@ wcstring process_iterator_t::name_for_pid(pid_t pid) { } args = (char *)malloc(maxarg); - if (!args) return result; + if (args == NULL) { // cppcheck-suppress memleak + return result; + } mib[0] = CTL_KERN; mib[1] = KERN_PROCARGS2; @@ -1582,7 +1584,7 @@ bool fish_xdm_login_hack_hack_hack_hack(std::vector *cmds, int argc } std::map abbreviations; -void update_abbr_cache(const wchar_t *op, const wcstring &varname) { +void update_abbr_cache(const wchar_t *op, const wcstring varname) { wcstring abbr; if (!unescape_string(varname.substr(wcslen(L"_fish_abbr_")), &abbr, 0, STRING_STYLE_VAR)) { debug(1, L"Abbreviation var '%ls' is not correctly encoded, ignoring it.", varname.c_str()); diff --git a/src/expand.h b/src/expand.h index 20181a59f..6643b8742 100644 --- a/src/expand.h +++ b/src/expand.h @@ -134,7 +134,7 @@ wcstring replace_home_directory_with_tilde(const wcstring &str); /// Abbreviation support. Expand src as an abbreviation, returning true if one was found, false if /// not. If result is not-null, returns the abbreviation by reference. -void update_abbr_cache(const wchar_t *op, const wcstring &varname); +void update_abbr_cache(const wchar_t *op, const wcstring varnam); bool expand_abbreviation(const wcstring &src, wcstring *output); // Terrible hacks diff --git a/src/fish.cpp b/src/fish.cpp index a3f45dfe3..096805a6e 100644 --- a/src/fish.cpp +++ b/src/fish.cpp @@ -421,7 +421,7 @@ int main(int argc, char **argv) { list.push_back(str2wcstring(*ptr)); } auto val = list_to_array_val(list); - env_set(L"argv", val->c_str(), ENV_DEFAULT); + env_set(L"argv", *val == ENV_NULL ? NULL : val->c_str(), 0); const wcstring rel_filename = str2wcstring(file); From 5786f9e5c38cd06ec76b368caab749f62713ad38 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 14:58:00 -0700 Subject: [PATCH 29/79] Revert "replace `var_entry_t` with `env_var_t`" This reverts commit 1c9370dbd2a3a0135c9c5fdd6b2a690dcd0c8027. --- src/builtin_set.cpp | 2 +- src/env.cpp | 120 +++++++++++++++++++---------------- src/env.h | 25 ++++---- src/env_universal_common.cpp | 42 ++++++------ src/env_universal_common.h | 4 +- src/fish_tests.cpp | 7 +- src/highlight.cpp | 4 +- src/path.cpp | 8 ++- tests/read.err | 3 - tests/read.in | 28 ++++---- tests/read.out | 3 - 11 files changed, 126 insertions(+), 120 deletions(-) diff --git a/src/builtin_set.cpp b/src/builtin_set.cpp index 810451033..cad1034ae 100644 --- a/src/builtin_set.cpp +++ b/src/builtin_set.cpp @@ -296,7 +296,7 @@ static int my_env_set(const wchar_t *cmd, const wchar_t *key, const wcstring_lis // We don't check `val->empty()` because an array var with a single empty string will be // "empty". A truly empty array var is set to the special value `ENV_NULL`. auto val = list_to_array_val(list); - retval = env_set(key, val->c_str(), scope | ENV_USER); + retval = env_set(key, *val == ENV_NULL ? NULL : val->c_str(), scope | ENV_USER); switch (retval) { case ENV_OK: { retval = STATUS_CMD_OK; diff --git a/src/env.cpp b/src/env.cpp index 63389d4ed..736a73433 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -126,8 +126,8 @@ class env_node_t { /// Pointer to next level. std::unique_ptr next; - /// Returns the given entry if present else env_var_t::missing_var. - const env_var_t find_entry(const wcstring &key); + /// Returns a pointer to the given entry if present, or NULL. + const var_entry_t *find_entry(const wcstring &key); }; class variable_entry_t { @@ -174,21 +174,27 @@ struct var_stack_t { env_node_t *next_scope_to_search(env_node_t *node); const env_node_t *next_scope_to_search(const env_node_t *node) const; - // Returns the scope used for unspecified scopes. An unspecified scope is either the topmost - // shadowing scope, or the global scope if none. This implements the default behavior of `set`. + // Returns the scope used for unspecified scopes + // An unspecified scope is either the topmost shadowing scope, or the global scope if none + // This implements the default behavior of 'set' env_node_t *resolve_unspecified_scope(); }; void var_stack_t::push(bool new_scope) { std::unique_ptr node(new env_node_t(new_scope)); - // Copy local-exported variables. + // Copy local-exported variables auto top_node = top.get(); - // Only if we introduce a new shadowing scope; i.e. not if it's just `begin; end` or - // "--no-scope-shadowing". - if (new_scope && top_node != this->global_env) { - for (auto &var : top_node->env) { - if (var.second.exportv) node->env.insert(var); + // Only if we introduce a new shadowing scope + // i.e. not if it's just `begin; end` or "--no-scope-shadowing". + if (new_scope) { + if (!(top_node == this->global_env)) { + for (auto &var : top_node->env) { + if (var.second.exportv) { + // This should copy var + node->env.insert(var); + } + } } } @@ -234,8 +240,8 @@ void var_stack_t::pop() { assert(this->top != NULL); for (const auto &entry_pair : old_top->env) { - const env_var_t &var = entry_pair.second; - if (var.exportv) { + const var_entry_t &entry = entry_pair.second; + if (entry.exportv) { this->mark_changed_exported(); break; } @@ -303,10 +309,13 @@ static bool is_electric(const wcstring &key) { return env_electric.find(key.c_str()) != env_electric.end(); } -const env_var_t env_node_t::find_entry(const wcstring &key) { - var_table_t::const_iterator entry = env.find(key); - if (entry != env.end()) return entry->second; - return env_var_t::missing_var(); +const var_entry_t *env_node_t::find_entry(const wcstring &key) { + const var_entry_t *result = NULL; + var_table_t::const_iterator where = env.find(key); + if (where != env.end()) { + result = &where->second; + } + return result; } /// Return the current umask value. @@ -950,8 +959,10 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { static env_node_t *env_get_node(const wcstring &key) { env_node_t *env = vars_stack().top.get(); while (env != NULL) { - env_var_t var = env->find_entry(key); - if (!var.missing()) break; + if (env->find_entry(key) != NULL) { + break; + } + env = vars_stack().next_scope_to_search(env); } return env; @@ -1048,8 +1059,8 @@ int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t var_mode) if (preexisting_node != NULL) { var_table_t::const_iterator result = preexisting_node->env.find(key); assert(result != preexisting_node->env.end()); - const env_var_t &var = result->second; - if (var.exportv) { + const var_entry_t &entry = result->second; + if (entry.exportv) { preexisting_entry_exportv = true; has_changed_new = true; } @@ -1063,7 +1074,7 @@ int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t var_mode) } else if (preexisting_node != NULL) { node = preexisting_node; if ((var_mode & (ENV_EXPORT | ENV_UNEXPORT)) == 0) { - // Use existing entry's exportv status. + // use existing entry's exportv var_mode = //!OCLINT(parameter reassignment) preexisting_entry_exportv ? ENV_EXPORT : 0; } @@ -1097,21 +1108,20 @@ int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t var_mode) if (!done) { // Set the entry in the node. Note that operator[] accesses the existing entry, or // creates a new one. - env_var_t &var = node->env[key]; - if (var.exportv) { + var_entry_t &entry = node->env[key]; + if (entry.exportv) { // This variable already existed, and was exported. has_changed_new = true; } - - var.set_val(val); + entry.val = val; if (var_mode & ENV_EXPORT) { // The new variable is exported. - var.exportv = true; + entry.exportv = true; node->exportv = true; has_changed_new = true; } else { - var.exportv = false; - // Set the node's exported when it changes something about exports + entry.exportv = false; + // Set the node's exportv when it changes something about exports // (also when it redefines a variable to not be exported). node->exportv = has_changed_old != has_changed_new; } @@ -1209,7 +1219,7 @@ int env_remove(const wcstring &key, int var_mode) { } wcstring env_var_t::as_string(void) const { - //assert(!is_missing); + assert(!is_missing); return val; } @@ -1266,15 +1276,12 @@ env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { env_node_t *env = search_local ? vars_stack().top.get() : vars_stack().global_env; while (env != NULL) { - const env_var_t var = env->find_entry(key); - if (!var.missing() && (var.exportv ? search_exported : search_unexported)) { - // The following statement is wrong because ENV_NULL is supposed to mean the var is - // set but has zero elements. Therefore we should not return missing_var. However, - // If this is changed to return `var` without the special-case then unit tests fail. - // Note that tokenize_variable_array() explicitly handles a var whose string - // representation contains only ENV_NULL. - if (var.as_string() == ENV_NULL) return env_var_t::missing_var(); - return var; + const var_entry_t *entry = env->find_entry(key); + if (entry != NULL && (entry->exportv ? search_exported : search_unexported)) { + if (entry->val == ENV_NULL) { + return env_var_t::missing_var(); + } + return entry->val; } if (has_scope) { @@ -1335,8 +1342,8 @@ bool env_exist(const wchar_t *key, env_mode_flags_t mode) { var_table_t::const_iterator result = env->env.find(key); if (result != env->env.end()) { - const env_var_t var = result->second; - return var.exportv ? test_exported : test_unexported; + const var_entry_t &res = result->second; + return res.exportv ? test_exported : test_unexported; } env = vars_stack().next_scope_to_search(env); } @@ -1378,9 +1385,9 @@ static void add_key_to_string_set(const var_table_t &envs, std::set *s bool show_exported, bool show_unexported) { var_table_t::const_iterator iter; for (iter = envs.begin(); iter != envs.end(); ++iter) { - const env_var_t &var = iter->second; + const var_entry_t &e = iter->second; - if ((var.exportv && show_exported) || (!var.exportv && show_unexported)) { + if ((e.exportv && show_exported) || (!e.exportv && show_unexported)) { // Insert this key. str_set->insert(iter->first); } @@ -1433,39 +1440,39 @@ wcstring_list_t env_get_names(int flags) { } /// Get list of all exported variables. -static void get_exported(const env_node_t *n, var_table_t &h) { +static void get_exported(const env_node_t *n, std::map *h) { if (!n) return; - if (n->new_scope) { + if (n->new_scope) get_exported(vars_stack().global_env, h); - } else { + else get_exported(n->next.get(), h); - } var_table_t::const_iterator iter; for (iter = n->env.begin(); iter != n->env.end(); ++iter) { const wcstring &key = iter->first; - const env_var_t var = iter->second; + const var_entry_t &val_entry = iter->second; - if (!var.missing() && var.exportv) { + if (val_entry.exportv && val_entry.val != ENV_NULL) { // Export the variable. Don't use std::map::insert here, since we need to overwrite // existing values from previous scopes. - h[key] = var; + (*h)[key] = val_entry.val; } else { // We need to erase from the map if we are not exporting, since a lower scope may have // exported. See #2132. - h.erase(key); + h->erase(key); } } } // Given a map from key to value, add values to out of the form key=value. -static void export_func(const var_table_t &envs, std::vector &out) { +static void export_func(const std::map &envs, std::vector &out) { out.reserve(out.size() + envs.size()); - for (auto iter = envs.begin(); iter != envs.end(); ++iter) { + std::map::const_iterator iter; + for (iter = envs.begin(); iter != envs.end(); ++iter) { const wcstring &key = iter->first; const std::string &ks = wcs2string(key); - std::string vs = wcs2string(iter->second.as_string()); + std::string vs = wcs2string(iter->second); // Arrays in the value are ASCII record separator (0x1e) delimited. But some variables // should have colons. Add those. @@ -1490,10 +1497,11 @@ void var_stack_t::update_export_array_if_necessary() { if (!this->has_changed_exported) { return; } + std::map vals; debug(4, L"env_export_arr() recalc"); - var_table_t vals; - get_exported(this->top.get(), vals); + + get_exported(this->top.get(), &vals); if (uvars()) { const wcstring_list_t uni = uvars()->get_names(true, false); @@ -1536,7 +1544,7 @@ void env_set_argv(const wchar_t *const *argv) { env_set(L"argv", sb.c_str(), ENV_LOCAL); } else { - env_set(L"argv", NULL, ENV_LOCAL); + env_set(L"argv", 0, ENV_LOCAL); } } diff --git a/src/env.h b/src/env.h index aee366f6c..b5c8131c5 100644 --- a/src/env.h +++ b/src/env.h @@ -75,19 +75,16 @@ class env_var_t { bool is_missing; public: - bool exportv; // whether the variable should be exported - static env_var_t missing_var() { env_var_t result((wcstring())); result.is_missing = true; - result.exportv = false; return result; } - env_var_t(const env_var_t &x) : val(x.val), is_missing(x.is_missing), exportv(x.exportv) {} - env_var_t(const wcstring &x) : val(x), is_missing(false), exportv(false) {} - env_var_t(const wchar_t *x) : val(x), is_missing(false), exportv(false) {} - env_var_t() : val(L""), is_missing(false), exportv(false) {} + env_var_t(const env_var_t &x) : val(x.val), is_missing(x.is_missing) {} + env_var_t(const wcstring &x) : val(x), is_missing(false) {} + env_var_t(const wchar_t *x) : val(x), is_missing(false) {} + env_var_t() : val(L""), is_missing(false) {} bool empty(void) const { return val.empty(); }; bool missing(void) const { return is_missing; } @@ -99,14 +96,10 @@ class env_var_t { env_var_t &operator=(const env_var_t &v) { is_missing = v.is_missing; - exportv = v.exportv; val = v.val; return *this; } - void set_val(const wcstring &s) { val = s; is_missing = false; } - void set_val(const wchar_t *s) { val = s; is_missing = false; } - bool operator==(const env_var_t &s) const { return is_missing == s.is_missing && val == s.val; } bool operator==(const wcstring &s) const { return !is_missing && val == s; } @@ -199,7 +192,15 @@ class env_vars_snapshot_t { extern int g_fork_count; extern bool g_use_posix_spawn; -typedef std::map var_table_t; +/// A variable entry. Stores the value of a variable and whether it should be exported. +struct var_entry_t { + wcstring val; // the value of the variable + bool exportv; // whether the variable should be exported + + var_entry_t() : exportv(false) {} +}; + +typedef std::map var_table_t; extern bool term_has_xn; // does the terminal have the "eat_newline_glitch" diff --git a/src/env_universal_common.cpp b/src/env_universal_common.cpp index 8284753c3..eac8b1e10 100644 --- a/src/env_universal_common.cpp +++ b/src/env_universal_common.cpp @@ -273,7 +273,7 @@ env_var_t env_universal_t::get(const wcstring &name) const { env_var_t result = env_var_t::missing_var(); var_table_t::const_iterator where = vars.find(name); if (where != vars.end()) { - result = where->second; + result = env_var_t(where->second.val); } return result; } @@ -295,10 +295,10 @@ void env_universal_t::set_internal(const wcstring &key, const wcstring &val, boo return; } - env_var_t &entry = vars[key]; - if (entry.exportv != exportv || entry.as_string() != val) { - entry.set_val(val); - entry.exportv = exportv; + var_entry_t *entry = &vars[key]; + if (entry->exportv != exportv || entry->val != val) { + entry->val = val; + entry->exportv = exportv; // If we are overwriting, then this is now modified. if (overwrite) { @@ -332,8 +332,8 @@ wcstring_list_t env_universal_t::get_names(bool show_exported, bool show_unexpor var_table_t::const_iterator iter; for (iter = vars.begin(); iter != vars.end(); ++iter) { const wcstring &key = iter->first; - const env_var_t &var = iter->second; - if ((var.exportv && show_exported) || (!var.exportv && show_unexported)) { + const var_entry_t &e = iter->second; + if ((e.exportv && show_exported) || (!e.exportv && show_unexported)) { result.push_back(key); } } @@ -369,18 +369,18 @@ void env_universal_t::generate_callbacks(const var_table_t &new_vars, } // See if the value has changed. - const env_var_t &new_entry = iter->second; + const var_entry_t &new_entry = iter->second; var_table_t::const_iterator existing = this->vars.find(key); if (existing == this->vars.end() || existing->second.exportv != new_entry.exportv || - existing->second != new_entry) { + existing->second.val != new_entry.val) { // Value has changed. callbacks.push_back( - callback_data_t(new_entry.exportv ? SET_EXPORT : SET, key, new_entry.as_string())); + callback_data_t(new_entry.exportv ? SET_EXPORT : SET, key, new_entry.val)); } } } -void env_universal_t::acquire_variables(var_table_t &vars_to_acquire) { +void env_universal_t::acquire_variables(var_table_t *vars_to_acquire) { // Copy modified values from existing vars to vars_to_acquire. for (std::set::iterator iter = this->modified.begin(); iter != this->modified.end(); ++iter) { @@ -388,19 +388,19 @@ void env_universal_t::acquire_variables(var_table_t &vars_to_acquire) { var_table_t::iterator src_iter = this->vars.find(key); if (src_iter == this->vars.end()) { /* The value has been deleted. */ - vars_to_acquire.erase(key); + vars_to_acquire->erase(key); } else { // The value has been modified. Copy it over. Note we can destructively modify the // source entry in vars since we are about to get rid of this->vars entirely. - env_var_t &src = src_iter->second; - env_var_t &dst = vars_to_acquire[key]; - dst.set_val(src.as_string()); + var_entry_t &src = src_iter->second; + var_entry_t &dst = (*vars_to_acquire)[key]; + dst.val = std::move(src.val); dst.exportv = src.exportv; } } // We have constructed all the callbacks and updated vars_to_acquire. Acquire it! - this->vars = std::move(vars_to_acquire); + this->vars = std::move(*vars_to_acquire); } void env_universal_t::load_from_fd(int fd, callback_data_list_t &callbacks) { @@ -418,7 +418,7 @@ void env_universal_t::load_from_fd(int fd, callback_data_list_t &callbacks) { this->generate_callbacks(new_vars, callbacks); // Acquire the new variables. - this->acquire_variables(new_vars); + this->acquire_variables(&new_vars); last_read_file = current_file; } } @@ -464,8 +464,8 @@ bool env_universal_t::write_to_fd(int fd, const wcstring &path) { // Append the entry. Note that append_file_entry may fail, but that only affects one // variable; soldier on. const wcstring &key = iter->first; - const env_var_t &var = iter->second; - append_file_entry(var.exportv ? SET_EXPORT : SET, key, var.as_string(), &contents, &storage); + const var_entry_t &entry = iter->second; + append_file_entry(entry.exportv ? SET_EXPORT : SET, key, entry.val, &contents, &storage); // Go to next. ++iter; @@ -845,9 +845,9 @@ void env_universal_t::parse_message_internal(const wcstring &msgstr, var_table_t wcstring val; if (unescape_string(tmp + 1, &val, 0)) { - env_var_t &entry = (*vars)[key]; + var_entry_t &entry = (*vars)[key]; entry.exportv = exportv; - entry.set_val(val); // acquire the value + entry.val = std::move(val); // acquire the value } } else { debug(1, PARSE_ERR, msg); diff --git a/src/env_universal_common.h b/src/env_universal_common.h index bca672ac7..a73a57c43 100644 --- a/src/env_universal_common.h +++ b/src/env_universal_common.h @@ -60,9 +60,9 @@ class env_universal_t { // the new vars. void generate_callbacks(const var_table_t &new_vars, callback_data_list_t &callbacks) const; - // Given a variable table, copy unmodified values into self. May destructively modify + // Given a variable table, copy unmodified values into self. May destructively modified // vars_to_acquire. - void acquire_variables(var_table_t &vars_to_acquire); + void acquire_variables(var_table_t *vars_to_acquire); static void parse_message_internal(const wcstring &msg, var_table_t *vars, wcstring *storage); static var_table_t read_message_internal(int fd); diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 9c539d96e..09e57e091 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -215,7 +215,6 @@ static void popd() { /// Test that the fish functions for converting strings to numbers work. static void test_str_to_num() { - say(L"Testing str_to_num"); const wchar_t *end; int i; long l; @@ -4369,9 +4368,6 @@ int main(int argc, char **argv) { // Set default signal handlers, so we can ctrl-C out of this. signal_reset_handlers(); - if (should_test_function("utility_functions")) test_utility_functions(); - if (should_test_function("wcstring_tok")) test_wcstring_tok(); - if (should_test_function("env_vars")) test_env_vars(); if (should_test_function("str_to_num")) test_str_to_num(); if (should_test_function("highlighting")) test_highlighting(); if (should_test_function("new_parser_ll2")) test_new_parser_ll2(); @@ -4413,12 +4409,15 @@ int main(int argc, char **argv) { if (should_test_function("autosuggestion_ignores")) test_autosuggestion_ignores(); if (should_test_function("autosuggestion_combining")) test_autosuggestion_combining(); if (should_test_function("autosuggest_suggest_special")) test_autosuggest_suggest_special(); + if (should_test_function("wcstring_tok")) test_wcstring_tok(); if (should_test_function("history")) history_tests_t::test_history(); if (should_test_function("history_merge")) history_tests_t::test_history_merge(); if (should_test_function("history_races")) history_tests_t::test_history_races(); if (should_test_function("history_formats")) history_tests_t::test_history_formats(); if (should_test_function("string")) test_string(); + if (should_test_function("env_vars")) test_env_vars(); if (should_test_function("illegal_command_exit_code")) test_illegal_command_exit_code(); + if (should_test_function("utility_functions")) test_utility_functions(); // history_tests_t::test_history_speed(); say(L"Encountered %d errors in low-level tests", err_count); diff --git a/src/highlight.cpp b/src/highlight.cpp index aefafa436..2a64336f9 100644 --- a/src/highlight.cpp +++ b/src/highlight.cpp @@ -279,6 +279,7 @@ rgb_color_t highlight_get_color(highlight_spec_t highlight, bool is_background) // Handle modifiers. if (highlight & highlight_modifier_valid_path) { env_var_t var2 = env_get(L"fish_color_valid_path"); + const wcstring val2 = var2.missing() ? L"" : var2.c_str(); rgb_color_t result2 = parse_color(var2, is_background); if (result.is_normal()) @@ -1013,9 +1014,8 @@ static bool command_is_valid(const wcstring &cmd, enum parse_statement_decoratio if (!is_valid && command_ok) is_valid = path_get_path(cmd, NULL, vars); // Implicit cd - if (!is_valid && implicit_cd_ok) { + if (!is_valid && implicit_cd_ok) is_valid = path_can_be_implicit_cd(cmd, NULL, working_directory.c_str(), vars); - } // Return what we got. return is_valid; diff --git a/src/path.cpp b/src/path.cpp index 70391067d..40ecd0757 100644 --- a/src/path.cpp +++ b/src/path.cpp @@ -48,18 +48,20 @@ static bool path_get_path_core(const wcstring &cmd, wcstring *out_path, } int err = ENOENT; - wcstring_list_t pathsv; + wcstring bin_path; if (!bin_path_var.missing()) { - bin_path_var.to_list(pathsv); + bin_path = bin_path_var.as_string(); } else { // Note that PREFIX is defined in the `Makefile` and is thus defined when this module is // compiled. This ensures we always default to "/bin", "/usr/bin" and the bin dir defined // for the fish programs. Possibly with a duplicate dir if PREFIX is empty, "/", "/usr" or // "/usr/". If the PREFIX duplicates /bin or /usr/bin that is harmless other than a trivial // amount of time testing a path we've already tested. - pathsv = wcstring_list_t({L"/bin", L"/usr/bin", PREFIX L"/bin"}); + bin_path = *list_to_array_val(wcstring_list_t({L"/bin", L"/usr/bin", PREFIX L"/bin"})); } + wcstring_list_t pathsv; + bin_path_var.to_list(pathsv); for (auto next_path : pathsv) { if (next_path.empty()) continue; append_path_component(next_path, cmd); diff --git a/tests/read.err b/tests/read.err index d76028bc0..676c6e2d1 100644 --- a/tests/read.err +++ b/tests/read.err @@ -8,9 +8,6 @@ read: Expected at least 1 args, got only 0 read: Expected 1 args, got 0 read: Expected 1 args, got 2 -#################### -# Verify correct behavior of subcommands and splitting of input. - #################### # Test splitting input diff --git a/tests/read.in b/tests/read.in index cd5bfa07e..36cbcc98e 100644 --- a/tests/read.in +++ b/tests/read.in @@ -3,6 +3,8 @@ # Test read builtin and IFS. # +########### +# Start by testing that invocation errors are handled correctly. logmsg Read with no vars is an error read @@ -11,19 +13,19 @@ read -a read -a v1 v2 read -a v1 -logmsg Verify correct behavior of subcommands and splitting of input. -begin - count (echo one\ntwo) - set -l IFS \t - count (echo one\ntwo) - set -l IFS - count (echo one\ntwo) - echo [(echo -n one\ntwo)] - count (echo one\ntwo\n) - echo [(echo -n one\ntwo\n)] - count (echo one\ntwo\n\n) - echo [(echo -n one\ntwo\n\n)] -end +########### +# Verify correct behavior of subcommands and splitting of input. +count (echo one\ntwo) +set -l IFS \t +count (echo one\ntwo) +set -l IFS +count (echo one\ntwo) +echo [(echo -n one\ntwo)] +count (echo one\ntwo\n) +echo [(echo -n one\ntwo\n)] +count (echo one\ntwo\n\n) +echo [(echo -n one\ntwo\n\n)] +set -le IFS function print_vars --no-scope-shadowing set -l space diff --git a/tests/read.out b/tests/read.out index eb1c23a6d..4b495bdc3 100644 --- a/tests/read.out +++ b/tests/read.out @@ -4,9 +4,6 @@ #################### # Read with -a and anything other than exactly on var name is an error - -#################### -# Verify correct behavior of subcommands and splitting of input. 2 2 1 From b0c47c814fc7ec126267223739fc35f3501bb757 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:22:50 -0700 Subject: [PATCH 30/79] Revert "make style-all time again" This reverts commit 975a5bfbdeeee59b6c46febe9de4d00a51f68196. It was meant for the major branch. --- src/complete.cpp | 13 +++++++------ src/env.cpp | 3 +-- src/exec.cpp | 30 ++++++++++++++++-------------- src/input_common.cpp | 4 +++- src/postfork.cpp | 22 ++++++++++++---------- src/postfork.h | 4 ++-- src/proc.cpp | 33 ++++++++++++++++++--------------- src/signal.cpp | 4 +++- src/wcstringutil.h | 2 +- 9 files changed, 63 insertions(+), 52 deletions(-) diff --git a/src/complete.cpp b/src/complete.cpp index 53b4f0954..2df3cb501 100644 --- a/src/complete.cpp +++ b/src/complete.cpp @@ -19,8 +19,8 @@ #include #include #include -#include #include +#include #include "autoload.h" #include "builtin.h" @@ -248,7 +248,9 @@ static Iterator unique_unsorted(Iterator begin, Iterator end, HashFunction hash) typedef typename std::iterator_traits::value_type T; std::unordered_set temp; - return std::remove_if(begin, end, [&](const T &val) { return !temp.insert(hash(val)).second; }); + return std::remove_if(begin, end, [&](const T& val) { + return !temp.insert(hash(val)).second; + }); } void completions_sort_and_prioritize(std::vector *comps) { @@ -274,10 +276,9 @@ void completions_sort_and_prioritize(std::vector *comps) { // Sort, provided COMPLETION_DONT_SORT isn't set stable_sort(comps->begin(), comps->end(), completion_t::is_naturally_less_than); // Deduplicate both sorted and unsorted results - comps->erase( - unique_unsorted(comps->begin(), comps->end(), - [](const completion_t &c) { return std::hash{}(c.completion); }), - comps->end()); + comps->erase(unique_unsorted(comps->begin(), comps->end(), [](const completion_t &c) { + return std::hash{}(c.completion); + }), comps->end()); // Sort the remainder by match type. They're already sorted alphabetically. stable_sort(comps->begin(), comps->end(), compare_completions_by_match_type); diff --git a/src/env.cpp b/src/env.cpp index 736a73433..f3db6495a 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -929,8 +929,7 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { // Set g_use_posix_spawn. Default to true. env_var_t use_posix_spawn = env_get(L"fish_use_posix_spawn"); - g_use_posix_spawn = - use_posix_spawn.missing_or_empty() ? true : from_string(use_posix_spawn.as_string()); + g_use_posix_spawn = use_posix_spawn.missing_or_empty() ? true : from_string(use_posix_spawn.as_string()); // Set fish_bind_mode to "default". env_set(FISH_BIND_MODE_VAR, DEFAULT_BIND_MODE, ENV_GLOBAL); diff --git a/src/exec.cpp b/src/exec.cpp index dc452d3b3..6f4e705e1 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -395,6 +395,7 @@ void internal_exec(job_t *j, const io_chain_t &&all_ios) { } } + void exec_job(parser_t &parser, job_t *j) { pid_t pid = 0; @@ -639,9 +640,8 @@ void exec_job(parser_t &parser, job_t *j) { // This is the io_streams we pass to internal builtins. std::unique_ptr builtin_io_streams(new io_streams_t(stdout_read_limit)); - auto do_fork = [&j, &p, &pid, &exec_error, &process_net_io_chain, &block_child, - &child_forked](bool drain_threads, const char *fork_type, - std::function child_action) -> bool { + auto do_fork = [&j, &p, &pid, &exec_error, &process_net_io_chain, &block_child, &child_forked] + (bool drain_threads, const char *fork_type, std::function child_action) -> bool { pid = execute_fork(drain_threads); if (pid == 0) { // This is the child process. Setup redirections, print correct output to @@ -649,10 +649,9 @@ void exec_job(parser_t &parser, job_t *j) { p->pid = getpid(); blocked_pid = -1; child_set_group(j, p); - // Make child processes pause after executing setup_child_process() to give - // down-chain commands in the job a chance to join their process group and read - // their pipes. The process will be resumed when the next command in the chain is - // started. + // Make child processes pause after executing setup_child_process() to give down-chain + // commands in the job a chance to join their process group and read their pipes. + // The process will be resumed when the next command in the chain is started. if (block_child) { kill(p->pid, SIGSTOP); } @@ -660,7 +659,8 @@ void exec_job(parser_t &parser, job_t *j) { setup_child_process(p, process_net_io_chain); child_action(); DIE("Child process returned control to do_fork lambda!"); - } else { + } + else { if (pid < 0) { debug(1, L"Failed to fork %s!\n", fork_type); job_mark_process_as_failed(j, p); @@ -669,8 +669,7 @@ void exec_job(parser_t &parser, job_t *j) { } // This is the parent process. Store away information on the child, and // possibly give it control over the terminal. - debug(2, L"Fork #%d, pid %d: %s for '%ls'", g_fork_count, pid, fork_type, - p->argv0()); + debug(2, L"Fork #%d, pid %d: %s for '%ls'", g_fork_count, pid, fork_type, p->argv0()); child_forked = true; if (block_child) { debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); @@ -684,7 +683,7 @@ void exec_job(parser_t &parser, job_t *j) { // Helper routine executed by INTERNAL_FUNCTION and INTERNAL_BLOCK_NODE to make sure an // output buffer exists in case there is another command in the job chain that will be // reading from this command's output. - auto verify_buffer_output = [&]() { + auto verify_buffer_output = [&] () { if (!p->is_last_in_job) { // Be careful to handle failure, e.g. too many open fds. block_output_io_buffer = io_buffer_t::create(STDOUT_FILENO, all_ios); @@ -701,6 +700,7 @@ void exec_job(parser_t &parser, job_t *j) { } }; + switch (p->type) { case INTERNAL_FUNCTION: { const wcstring func_name = p->argv0(); @@ -1082,14 +1082,16 @@ void exec_job(parser_t &parser, job_t *j) { if (pid == 0) { job_mark_process_as_failed(j, p); exec_error = true; - } else { + } + else { child_spawned = true; } } else #endif { - if (!do_fork(false, "external command", - [&] { safe_launch_process(p, actual_cmd, argv, envv); })) { + if (!do_fork(false, "external command", [&] { + safe_launch_process(p, actual_cmd, argv, envv); + })) { break; } } diff --git a/src/input_common.cpp b/src/input_common.cpp index 9cb994fe8..22c402a7d 100644 --- a/src/input_common.cpp +++ b/src/input_common.cpp @@ -58,7 +58,9 @@ static wint_t lookahead_front(void) { return lookahead_list.front(); } /// Callback function for handling interrupts on reading. static int (*interrupt_handler)(); -void input_common_init(int (*ih)()) { interrupt_handler = ih; } +void input_common_init(int (*ih)()) { + interrupt_handler = ih; +} void input_common_destroy() {} diff --git a/src/postfork.cpp b/src/postfork.cpp index f6d597ded..bdf788607 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -119,15 +119,15 @@ bool child_set_group(job_t *j, process_t *p) { return retval; } -/// Called only by the parent only after a child forks and successfully calls child_set_group, -/// guaranteeing the job control process group has been created and that the child belongs to the -/// correct process group. Here we can update our job_t structure to reflect the correct process -/// group in the case of JOB_CONTROL, and we can give the new process group control of the terminal -/// if it's to run in the foreground. Note that we can guarantee the child won't try to read from -/// the terminal before we've had a chance to run this code, because we haven't woken them up with a -/// SIGCONT yet. This musn't be called as a part of setup_child_process because that can hang -/// indefinitely until data is available to read/write in the case of IO_FILE, which means we'll -/// never reach our SIGSTOP and everything hangs. +/// Called only by the parent only after a child forks and successfully calls child_set_group, guaranteeing +/// the job control process group has been created and that the child belongs to the correct process group. +/// Here we can update our job_t structure to reflect the correct process group in the case of JOB_CONTROL, +/// and we can give the new process group control of the terminal if it's to run in the foreground. Note that +/// we can guarantee the child won't try to read from the terminal before we've had a chance to run this code, +/// because we haven't woken them up with a SIGCONT yet. +/// This musn't be called as a part of setup_child_process because that can hang indefinitely until data is +/// available to read/write in the case of IO_FILE, which means we'll never reach our SIGSTOP and everything +/// hangs. bool set_child_group(job_t *j, pid_t child_pid) { bool retval = true; @@ -148,7 +148,9 @@ bool set_child_group(job_t *j, pid_t child_pid) { // a process group, attempting to call tcsetpgrp from the background will cause SIGTTOU // to be sent to everything in our process group (unless we handle it). debug(4, L"Process group %d already has control of terminal\n", j->pgid); - } else { + } + else + { // No need to duplicate the code here, a function already exists that does just this. retval = terminal_give_to_job(j, false /*new job, so not continuing*/); } diff --git a/src/postfork.h b/src/postfork.h index c397e342e..f29be4741 100644 --- a/src/postfork.h +++ b/src/postfork.h @@ -18,8 +18,8 @@ class io_chain_t; class job_t; class process_t; -bool set_child_group(job_t *j, pid_t child_pid); // called by parent -bool child_set_group(job_t *j, process_t *p); // called by child +bool set_child_group(job_t *j, pid_t child_pid); //called by parent +bool child_set_group(job_t *j, process_t *p); //called by child /// Initialize a new child process. This should be called right away after forking in the child /// process. If job control is enabled for this job, the process is put in the process group of the diff --git a/src/proc.cpp b/src/proc.cpp index 7b6a630da..3f005b34f 100644 --- a/src/proc.cpp +++ b/src/proc.cpp @@ -803,10 +803,9 @@ bool terminal_give_to_job(job_t *j, int cont) { // been done. if (tcgetpgrp(STDIN_FILENO) == j->pgid) { debug(2, L"Process group %d already has control of terminal\n", j->pgid); - } else { - debug(4, - L"Attempting to bring process group to foreground via tcsetpgrp for job->pgid %d\n", - j->pgid); + } + else { + debug(4, L"Attempting to bring process group to foreground via tcsetpgrp for job->pgid %d\n", j->pgid); // The tcsetpgrp(2) man page says that EPERM is thrown if "pgrp has a supported value, but // is not the process group ID of a process in the same session as the calling process." @@ -820,13 +819,15 @@ bool terminal_give_to_job(job_t *j, int cont) { while (tcsetpgrp(STDIN_FILENO, j->pgid) != 0) { bool pgroup_terminated = false; if (errno == EINTR) { - ; // Always retry on EINTR, see comments in tcsetattr EINTR code below. - } else if (errno == EINVAL) { + ; // Always retry on EINTR, see comments in tcsetattr EINTR code below. + } + else if (errno == EINVAL) { // OS X returns EINVAL if the process group no longer lives. Probably other OSes, // too. Unlike EPERM below, EINVAL can only happen if the process group has // terminated. pgroup_terminated = true; - } else if (errno == EPERM) { + } + else if (errno == EPERM) { // Retry so long as this isn't because the process group is dead. int wait_result = waitpid(-1 * j->pgid, &wait_result, WNOHANG); if (wait_result == -1) { @@ -836,15 +837,16 @@ bool terminal_give_to_job(job_t *j, int cont) { // would mean processes from the group still exist but is still running in some // state or the other. pgroup_terminated = true; - } else { + } + else { // Debug the original tcsetpgrp error (not the waitpid errno) to the log, and // then retry until not EPERM or the process group has exited. debug(2, L"terminal_give_to_job(): EPERM.\n", j->pgid); } - } else { + } + else { if (errno == ENOTTY) redirect_tty_output(); - debug(1, _(L"Could not send job %d ('%ls') with pgid %d to foreground"), j->job_id, - j->command_wcstr(), j->pgid); + debug(1, _(L"Could not send job %d ('%ls') with pgid %d to foreground"), j->job_id, j->command_wcstr(), j->pgid); wperror(L"tcsetpgrp"); signal_unblock(); return false; @@ -872,8 +874,7 @@ bool terminal_give_to_job(job_t *j, int cont) { } if (result == -1) { if (errno == ENOTTY) redirect_tty_output(); - debug(1, _(L"Could not send job %d ('%ls') to foreground"), j->job_id, - j->command_wcstr()); + debug(1, _(L"Could not send job %d ('%ls') to foreground"), j->job_id, j->command_wcstr()); wperror(L"tcsetattr"); signal_unblock(); return false; @@ -935,8 +936,10 @@ void job_continue(job_t *j, bool cont) { j->set_flag(JOB_NOTIFIED, false); CHECK_BLOCK(); - debug(4, L"%ls job %d, gid %d (%ls), %ls, %ls", cont ? L"Continue" : L"Start", j->job_id, - j->pgid, j->command_wcstr(), job_is_completed(j) ? L"COMPLETED" : L"UNCOMPLETED", + debug(4, L"%ls job %d, gid %d (%ls), %ls, %ls", + cont ? L"Continue" : L"Start", + j->job_id, j->pgid, j->command_wcstr(), + job_is_completed(j) ? L"COMPLETED" : L"UNCOMPLETED", is_interactive ? L"INTERACTIVE" : L"NON-INTERACTIVE"); if (!job_is_completed(j)) { diff --git a/src/signal.cpp b/src/signal.cpp index cc8726e74..f36847b59 100644 --- a/src/signal.cpp +++ b/src/signal.cpp @@ -417,4 +417,6 @@ void signal_unblock() { } } -bool signal_is_blocked() { return static_cast(block_count); } +bool signal_is_blocked() { + return static_cast(block_count); +} diff --git a/src/wcstringutil.h b/src/wcstringutil.h index 04a1a4133..a9c03385d 100644 --- a/src/wcstringutil.h +++ b/src/wcstringutil.h @@ -28,7 +28,7 @@ wcstring_range wcstring_tok(wcstring& str, const wcstring& needle, /// If the needle is empty, split on individual elements (characters). template void split_about(ITER haystack_start, ITER haystack_end, ITER needle_start, ITER needle_end, - wcstring_list_t* output, long max) { + wcstring_list_t *output, long max) { long remaining = max; ITER haystack_cursor = haystack_start; while (remaining > 0 && haystack_cursor != haystack_end) { From dc5d0ff22f503af59bdc9eab95fd19f8f39fa3bf Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:23:19 -0700 Subject: [PATCH 31/79] Revert "lint and style cleanups" This reverts commit acdb81bbca0090fd5f13d72fa1c936c2b645fc42. It was meant for the major branch. --- src/env.h | 2 +- src/exec.cpp | 82 +++++++++++++++++++++++------------------------- src/expand.h | 2 ++ src/postfork.cpp | 22 ++++++------- src/postfork.h | 1 + src/proc.cpp | 66 ++++++++++++++++++-------------------- 6 files changed, 86 insertions(+), 89 deletions(-) diff --git a/src/env.h b/src/env.h index b5c8131c5..b7f3c2158 100644 --- a/src/env.h +++ b/src/env.h @@ -71,8 +71,8 @@ void misc_init(); class env_var_t { private: - wcstring val; bool is_missing; + wcstring val; public: static env_var_t missing_var() { diff --git a/src/exec.cpp b/src/exec.cpp index 6f4e705e1..0b18bb634 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -19,7 +19,6 @@ #include #include -#include #include #include #include @@ -38,7 +37,6 @@ #include "postfork.h" #include "proc.h" #include "reader.h" -#include "signal.h" #include "wutil.h" // IWYU pragma: keep /// File descriptor redirection error message. @@ -495,8 +493,8 @@ void exec_job(parser_t &parser, job_t *j) { // We are careful to set these to -1 when closed, so if we exit the loop abruptly, we can still // close them. bool pgrp_set = false; - // This is static since processes can block on input/output across jobs the main exec_job loop - // is only ever run in a single thread, so this is OK. + //this is static since processes can block on input/output across jobs + //the main exec_job loop is only ever run in a single thread, so this is OK static pid_t blocked_pid = -1; int pipe_current_read = -1, pipe_current_write = -1, pipe_next_read = -1; for (std::unique_ptr &unique_p : j->processes) { @@ -515,7 +513,7 @@ void exec_job(parser_t &parser, job_t *j) { // See if we need a pipe. const bool pipes_to_next_command = !p->is_last_in_job; - // Set to true if we end up forking for this process. + //set to true if we end up forking for this process bool child_forked = false; bool child_spawned = false; bool block_child = true; @@ -624,12 +622,12 @@ void exec_job(parser_t &parser, job_t *j) { // a pipeline. shared_ptr block_output_io_buffer; - // Used to SIGCONT the previously SIGSTOP'd process so the main loop or next command in job - // can read from its output. - auto unblock_previous = [&j]() { - // We've already called waitpid after forking the child, so we've guaranteed that - // they're already SIGSTOP'd. Otherwise we'd be risking a deadlock because we can call - // SIGCONT before they've actually stopped, and they'll remain suspended indefinitely. + //used to SIGCONT the previously SIGSTOP'd process so the main loop or next command in job can read + //from its output. + auto unblock_previous = [&j] () { + //we've already called waitpid after forking the child, so we've guaranteed that they're + //already SIGSTOP'd. Otherwise we'd be risking a deadlock because we can call SIGCONT before + //they've actually stopped, and they'll remain suspended indefinitely. if (blocked_pid != -1) { debug(2, L"Unblocking process %d.\n", blocked_pid); kill(blocked_pid, SIGCONT); @@ -655,7 +653,7 @@ void exec_job(parser_t &parser, job_t *j) { if (block_child) { kill(p->pid, SIGSTOP); } - // The parent will wake us up when we're ready to execute. + //the parent will wake us up when we're ready to execute setup_child_process(p, process_net_io_chain); child_action(); DIE("Child process returned control to do_fork lambda!"); @@ -680,9 +678,10 @@ void exec_job(parser_t &parser, job_t *j) { return true; }; - // Helper routine executed by INTERNAL_FUNCTION and INTERNAL_BLOCK_NODE to make sure an - // output buffer exists in case there is another command in the job chain that will be - // reading from this command's output. + + //helper routine executed by INTERNAL_FUNCTION and INTERNAL_BLOCK_NODE to make sure + //an output buffer exists in case there is another command in the job chain that will + //be reading from this command's output. auto verify_buffer_output = [&] () { if (!p->is_last_in_job) { // Be careful to handle failure, e.g. too many open fds. @@ -843,8 +842,8 @@ void exec_job(parser_t &parser, job_t *j) { const int fg = j->get_flag(JOB_FOREGROUND); j->set_flag(JOB_FOREGROUND, false); - // Main loop may need to perform a blocking read from previous command's output. - // Make sure read source is not blocked. + //main loop may need to perform a blocking read from previous command's output. + //make sure read source is not blocked unblock_previous(); p->status = builtin_run(parser, p->get_argv(), *builtin_io_streams); @@ -940,8 +939,8 @@ void exec_job(parser_t &parser, job_t *j) { bool must_fork = redirection_is_to_real_file(stdout_io.get()) || redirection_is_to_real_file(stderr_io.get()); if (!must_fork && p->is_last_in_job) { - // We are handling reads directly in the main loop. Make sure source is - // unblocked. Note that we may still end up forking. + //we are handling reads directly in the main loop. Make sure source is unblocked. + //Note that we may still end up forking. unblock_previous(); const bool stdout_is_to_buffer = stdout_io && stdout_io->io_mode == IO_BUFFER; const bool no_stdout_output = stdout_buffer.empty(); @@ -1111,19 +1110,18 @@ void exec_job(parser_t &parser, job_t *j) { bool child_blocked = block_child && child_forked; if (child_blocked) { - // We have to wait to ensure the child has set their progress group and is in SIGSTOP - // state otherwise, we can later call SIGCONT before they call SIGSTOP and they'll be - // blocked indefinitely. The child is SIGSTOP'd and is guaranteed to have called - // child_set_group() at this point. but we only need to call set_child_group for the - // first process in the group. If needs_keepalive is set, this has already been called - // for the keepalive process. + //we have to wait to ensure the child has set their progress group and is in SIGSTOP state + //otherwise, we can later call SIGCONT before they call SIGSTOP and they'll be blocked indefinitely. + //The child is SIGSTOP'd and is guaranteed to have called child_set_group() at this point. + //but we only need to call set_child_group for the first process in the group. + //If needs_keepalive is set, this has already been called for the keepalive process pid_t pid_status{}; int result; while ((result = waitpid(p->pid, &pid_status, WUNTRACED)) == -1 && errno == EINTR) { - // This could be a superfluous interrupt or Ctrl+C at the terminal In all cases, it - // is OK to retry since the forking code above is specifically designed to never, - // ever hang/block in a child process before the SIGSTOP call is reached. - ; // do nothing + //This could be a superfluous interrupt or Ctrl+C at the terminal + //In all cases, it is OK to retry since the forking code above is specifically designed + //to never, ever hang/block in a child process before the SIGSTOP call is reached. + continue; } if (result == -1) { exec_error = true; @@ -1131,26 +1129,26 @@ void exec_job(parser_t &parser, job_t *j) { wperror(L"waitpid"); break; } - // We are not unblocking the child via SIGCONT just yet to give the next process a - // chance to open the pipes and join the process group. We only unblock the last process - // in the job chain because no one awaits it. + //we are not unblocking the child via SIGCONT just yet to give the next process a chance to open + //the pipes and join the process group. We only unblock the last process in the job chain because + //no one awaits it. } - // Regardless of whether the child blocked or not: only once per job, and only once we've - // actually executed an external command. + //regardless of whether the child blocked or not: + //only once per job, and only once we've actually executed an external command if ((child_spawned || child_forked) && !pgrp_set) { - // This should be called after waitpid if child_forked and pipes_to_next_command it can - // be called at any time if child_spawned. + //this should be called after waitpid if child_forked && pipes_to_next_command + //it can be called at any time if child_spawned set_child_group(j, p->pid); - // we can't rely on p->is_first_in_job because a builtin may have been the first. + //we can't rely on p->is_first_in_job because a builtin may have been the first pgrp_set = true; } - // If the command we ran _before_ this one was SIGSTOP'd to let this one catch up, unblock - // it now. this must be after the wait_pid on the process we just started, if any. + //if the command we ran _before_ this one was SIGSTOP'd to let this one catch up, unblock it now. + //this must be after the wait_pid on the process we just started, if any. unblock_previous(); if (child_blocked) { - // Store the newly-blocked command's PID so that it can be SIGCONT'd once the next - // process in the chain is started. That may be in this job or in another job. + //store the newly-blocked command's PID so that it can be SIGCONT'd once the next process + //in the chain is started. That may be in this job or in another job. blocked_pid = p->pid; } @@ -1166,7 +1164,7 @@ void exec_job(parser_t &parser, job_t *j) { pipe_current_write = -1; } - // Unblock the last process because there's no need for it to stay SIGSTOP'd for anything. + //unblock the last process because there's no need for it to stay SIGSTOP'd for anything if (p->is_last_in_job) { unblock_previous(); } diff --git a/src/expand.h b/src/expand.h index 6643b8742..2fd0423bd 100644 --- a/src/expand.h +++ b/src/expand.h @@ -122,6 +122,8 @@ bool expand_one(wcstring &inout_str, expand_flags_t flags, parse_error_list_t *e /// Convert the variable value to a human readable form, i.e. escape things, handle arrays, etc. /// Suitable for pretty-printing. +/// +/// \param in the value to escape wcstring expand_escape_variable(const env_var_t &var); /// Perform tilde expansion and nothing else on the specified string, which is modified in place. diff --git a/src/postfork.cpp b/src/postfork.cpp index bdf788607..8b79c4f7f 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -78,9 +78,9 @@ bool child_set_group(job_t *j, process_t *p) { if (j->pgid == -2) { j->pgid = p->pid; } - // Retry on EPERM because there's no way that a child cannot join an existing progress group - // because we are SIGSTOPing the previous job in the chain. Sometimes we have to try a few - // times to get the kernel to see the new group. (Linux 4.4.0) + //retry on EPERM because there's no way that a child cannot join an existing progress group + //because we are SIGSTOPing the previous job in the chain. Sometimes we have to try a few + //times to get the kernel to see the new group. (Linux 4.4.0) int failure = setpgid(p->pid, j->pgid); while (failure == -1 && (errno == EPERM || errno == EINTR)) { debug_safe(4, "Retrying setpgid in child process"); @@ -112,7 +112,7 @@ bool child_set_group(job_t *j, process_t *p) { retval = false; } } else { - // This is probably stays unused in the child. + //this is probably stays unused in the child j->pgid = getpgrp(); } @@ -142,16 +142,16 @@ bool set_child_group(job_t *j, pid_t child_pid) { if (j->get_flag(JOB_TERMINAL) && j->get_flag(JOB_FOREGROUND)) { //!OCLINT(early exit) if (tcgetpgrp(STDIN_FILENO) == j->pgid) { - // We've already assigned the process group control of the terminal when the first - // process in the job was started. There's no need to do so again, and on some platforms - // this can cause an EPERM error. In addition, if we've given control of the terminal to - // a process group, attempting to call tcsetpgrp from the background will cause SIGTTOU - // to be sent to everything in our process group (unless we handle it). + //we've already assigned the process group control of the terminal when the first process in the job + //was started. There's no need to do so again, and on some platforms this can cause an EPERM error. + //In addition, if we've given control of the terminal to a process group, attempting to call tcsetpgrp + //from the background will cause SIGTTOU to be sent to everything in our process group (unless we + //handle it).. debug(4, L"Process group %d already has control of terminal\n", j->pgid); } else { - // No need to duplicate the code here, a function already exists that does just this. + //no need to duplicate the code here, a function already exists that does just this retval = terminal_give_to_job(j, false /*new job, so not continuing*/); } } @@ -264,7 +264,7 @@ int setup_child_process(process_t *p, const io_chain_t &io_chain) { bool ok = true; if (ok) { - // In the case of IO_FILE, this can hang until data is available to read/write! + //In the case of IO_FILE, this can hang until data is available to read/write! ok = (0 == handle_child_io(io_chain)); if (p != 0 && !ok) { debug_safe(4, "handle_child_io failed in setup_child_process"); diff --git a/src/postfork.h b/src/postfork.h index f29be4741..4a3b47e4e 100644 --- a/src/postfork.h +++ b/src/postfork.h @@ -27,6 +27,7 @@ bool child_set_group(job_t *j, process_t *p); //called by child /// inside the exec function, which blocks all signals), and all IO redirections and other file /// descriptor actions are performed. /// +/// \param j the job to set up the IO for /// \param p the child process to set up /// \param io_chain the IO chain to use /// diff --git a/src/proc.cpp b/src/proc.cpp index 3f005b34f..b0ebdf893 100644 --- a/src/proc.cpp +++ b/src/proc.cpp @@ -791,56 +791,53 @@ bool terminal_give_to_job(job_t *j, int cont) { signal_block(); - // It may not be safe to call tcsetpgrp if we've already done so, as at that point we are no - // longer the controlling process group for the terminal and no longer have permission to set - // the process group that is in control, causing tcsetpgrp to return EPERM, even though that's - // not the documented behavior in tcsetpgrp(3), which instead says other bad things will happen - // (it says SIGTTOU will be sent to all members of the background *calling* process group, but - // it's more complicated than that, SIGTTOU may or may not be sent depending on the TTY - // configuration and whether or not signal handlers for SIGTTOU are installed. Read: - // http://curiousthing.org/sigttin-sigttou-deep-dive-linux In all cases, our goal here was just - // to hand over control of the terminal to this process group, which is a no-op if it's already - // been done. + //It may not be safe to call tcsetpgrp if we've already done so, as at that point we are no longer + //the controlling process group for the terminal and no longer have permission to set the process + //group that is in control, causing tcsetpgrp to return EPERM, even though that's not the documented + //behavior in tcsetpgrp(3), which instead says other bad things will happen (it says SIGTTOU will be + //sent to all members of the background *calling* process group, but it's more complicated than that, + //SIGTTOU may or may not be sent depending on the TTY configuration and whether or not signal handlers + //for SIGTTOU are installed. Read: http://curiousthing.org/sigttin-sigttou-deep-dive-linux + //In all cases, our goal here was just to hand over control of the terminal to this process group, + //which is a no-op if it's already been done. if (tcgetpgrp(STDIN_FILENO) == j->pgid) { debug(2, L"Process group %d already has control of terminal\n", j->pgid); } else { debug(4, L"Attempting to bring process group to foreground via tcsetpgrp for job->pgid %d\n", j->pgid); - // The tcsetpgrp(2) man page says that EPERM is thrown if "pgrp has a supported value, but - // is not the process group ID of a process in the same session as the calling process." - // Since we _guarantee_ that this isn't the case (the child calls setpgid before it calls - // SIGSTOP, and the child was created in the same session as us), it seems that EPERM is - // being thrown because of an caching issue - the call to tcsetpgrp isn't seeing the - // newly-created process group just yet. On this developer's test machine (WSL running Linux - // 4.4.0), EPERM does indeed disappear on retry. The important thing is that we can - // guarantee the process isn't going to exit while we wait (which would cause us to possibly - // block indefinitely). + //the tcsetpgrp(2) man page says that EPERM is thrown if "pgrp has a supported value, but is not the + //process group ID of a process in the same session as the calling process." + //Since we _guarantee_ that this isn't the case (the child calls setpgid before it calls SIGSTOP, and + //the child was created in the same session as us), it seems that EPERM is being thrown because of an + //caching issue - the call to tcsetpgrp isn't seeing the newly-created process group just yet. On this + //developer's test machine (WSL running Linux 4.4.0), EPERM does indeed disappear on retry. The important + //thing is that we can guarantee the process isn't going to exit while we wait (which would cause us to + //possibly block indefinitely). + while (tcsetpgrp(STDIN_FILENO, j->pgid) != 0) { bool pgroup_terminated = false; if (errno == EINTR) { - ; // Always retry on EINTR, see comments in tcsetattr EINTR code below. + //always retry on EINTR, see comments in tcsetattr EINTR code below. } else if (errno == EINVAL) { - // OS X returns EINVAL if the process group no longer lives. Probably other OSes, - // too. Unlike EPERM below, EINVAL can only happen if the process group has - // terminated. + //OS X returns EINVAL if the process group no longer lives. Probably other OSes, too. + //Unlike EPERM below, EINVAL can only happen if the process group has terminated pgroup_terminated = true; } else if (errno == EPERM) { - // Retry so long as this isn't because the process group is dead. + //retry so long as this isn't because the process group is dead int wait_result = waitpid(-1 * j->pgid, &wait_result, WNOHANG); if (wait_result == -1) { - // Note that -1 is technically an "error" for waitpid in the sense that an - // invalid argument was specified because no such process group exists any - // longer. This is the observed behavior on Linux 4.4.0. a "success" result - // would mean processes from the group still exist but is still running in some - // state or the other. + //Note that -1 is technically an "error" for waitpid in the sense that an invalid argument was specified + //because no such process group exists any longer. This is the observed behavior on Linux 4.4.0. + //a "success" result would mean processes from the group still exist but is still running in some state + //or the other. pgroup_terminated = true; } else { - // Debug the original tcsetpgrp error (not the waitpid errno) to the log, and - // then retry until not EPERM or the process group has exited. + //debug the original tcsetpgrp error (not the waitpid errno) to the log, and then retry until not EPERM + //or the process group has exited. debug(2, L"terminal_give_to_job(): EPERM.\n", j->pgid); } } @@ -853,10 +850,9 @@ bool terminal_give_to_job(job_t *j, int cont) { } if (pgroup_terminated) { - // All processes in the process group has exited. Since we force all child procs to - // SIGSTOP on startup, the only way that can happen is if the very last process in - // the group terminated, and didn't need to access the terminal, otherwise it would - // have hung waiting for terminal IO (SIGTTIN). We can ignore this. + //all processes in the process group has exited. Since we force all child procs to SIGSTOP on startup, + //the only way that can happen is if the very last process in the group terminated, and didn't need + //to access the terminal, otherwise it would have hung waiting for terminal IO (SIGTTIN). We can ignore this. debug(3, L"tcsetpgrp called but process group %d has terminated.\n", j->pgid); break; } From 8aca33b21f73553067949349fb40273a850e867e Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:23:44 -0700 Subject: [PATCH 32/79] Revert "fixes to job control changes" This reverts commit 083224d1c0076b43e9dbb080bc501ee275a9619d. It was meant for the major branch. --- src/exec.cpp | 14 +++++++------- src/proc.cpp | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 0b18bb634..18b96ffb0 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -374,15 +374,15 @@ void internal_exec(job_t *j, const io_chain_t &&all_ios) { // really make sense, so I'm not trying to fix it here. if (!setup_child_process(0, all_ios)) { // Decrement SHLVL as we're removing ourselves from the shell "stack". - const env_var_t shlvl_var = env_get(L"SHLVL", ENV_GLOBAL | ENV_EXPORT); - wcstring shlvl_str = L"0"; - if (!shlvl_var.missing()) { - long shlvl = fish_wcstol(shlvl_var.c_str()); - if (!errno && shlvl > 0) { - shlvl_str = to_string(shlvl - 1); + const env_var_t shlvl_str = env_get_string(L"SHLVL", ENV_GLOBAL | ENV_EXPORT); + wcstring nshlvl_str = L"0"; + if (!shlvl_str.missing()) { + long shlvl_i = fish_wcstol(shlvl_str.c_str()); + if (!errno && shlvl_i > 0) { + nshlvl_str = to_string(shlvl_i - 1); } } - env_set(L"SHLVL", shlvl_str.c_str(), ENV_GLOBAL | ENV_EXPORT); + env_set(L"SHLVL", nshlvl_str.c_str(), ENV_GLOBAL | ENV_EXPORT); // launch_process _never_ returns. launch_process_nofork(j->processes.front().get()); diff --git a/src/proc.cpp b/src/proc.cpp index b0ebdf893..07e00b618 100644 --- a/src/proc.cpp +++ b/src/proc.cpp @@ -845,7 +845,7 @@ bool terminal_give_to_job(job_t *j, int cont) { if (errno == ENOTTY) redirect_tty_output(); debug(1, _(L"Could not send job %d ('%ls') with pgid %d to foreground"), j->job_id, j->command_wcstr(), j->pgid); wperror(L"tcsetpgrp"); - signal_unblock(); + signal_unblock(true); return false; } From fea22f2bec82e9eaffa8a7ecf07115d394d3d26d Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:23:57 -0700 Subject: [PATCH 33/79] Revert "Revert "Revert "finish cleanup of signal blocking code""" This reverts commit 52d739c746034e0dcd554001f5927ad2f32a20fc. It was meant for the major branch. --- src/exec.cpp | 25 +++++++++++++++++++++++++ src/fish.cpp | 7 +++++++ src/history.cpp | 9 +++++++++ src/postfork.cpp | 2 ++ src/proc.cpp | 14 +++++++------- src/reader.cpp | 16 ++++++++++++++-- src/signal.cpp | 16 +++++++++++++--- src/signal.h | 6 ++++-- 8 files changed, 81 insertions(+), 14 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 18b96ffb0..891159005 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -37,6 +37,7 @@ #include "postfork.h" #include "proc.h" #include "reader.h" +#include "signal.h" #include "wutil.h" // IWYU pragma: keep /// File descriptor redirection error message. @@ -322,12 +323,16 @@ static void internal_exec_helper(parser_t &parser, const wcstring &def, node_off return; } + signal_unblock(); + if (node_offset == NODE_OFFSET_INVALID) { parser.eval(def, morphed_chain, block_type); } else { parser.eval_block_node(node_offset, morphed_chain, block_type); } + signal_block(); + morphed_chain.clear(); io_cleanup_fds(opened_fds); job_reap(0); @@ -449,6 +454,8 @@ void exec_job(parser_t &parser, job_t *j) { } } + signal_block(); + // See if we need to create a group keepalive process. This is a process that we create to make // sure that the process group doesn't die accidentally, and is often needed when a // builtin/block/function is inside a pipeline, since that usually means we have to wait for one @@ -702,6 +709,9 @@ void exec_job(parser_t &parser, job_t *j) { switch (p->type) { case INTERNAL_FUNCTION: { + // Calls to function_get_definition might need to source a file as a part of + // autoloading, hence there must be no blocks. + signal_unblock(); const wcstring func_name = p->argv0(); wcstring def; bool function_exists = function_get_definition(func_name, &def); @@ -709,6 +719,8 @@ void exec_job(parser_t &parser, job_t *j) { const std::map inherit_vars = function_get_inherit_vars(func_name); + signal_block(); + if (!function_exists) { debug(0, _(L"Unknown function '%ls'"), p->argv0()); break; @@ -716,7 +728,13 @@ void exec_job(parser_t &parser, job_t *j) { function_block_t *fb = parser.push_block(p, func_name, shadow_scope); + + // Setting variables might trigger an event handler, hence we need to unblock + // signals. + signal_unblock(); function_prepare_environment(func_name, p->get_argv() + 1, inherit_vars); + signal_block(); + parser.forbid_function(func_name); verify_buffer_output(); @@ -845,8 +863,13 @@ void exec_job(parser_t &parser, job_t *j) { //main loop may need to perform a blocking read from previous command's output. //make sure read source is not blocked unblock_previous(); + + signal_unblock(); + p->status = builtin_run(parser, p->get_argv(), *builtin_io_streams); + signal_block(); + // Restore the fg flag, which is temporarily set to false during builtin // execution so as not to confuse some job-handling builtins. j->set_flag(JOB_FOREGROUND, fg); @@ -1180,7 +1203,9 @@ void exec_job(parser_t &parser, job_t *j) { kill(keepalive.pid, SIGKILL); } + signal_unblock(); debug(3, L"Job is constructed"); + j->set_flag(JOB_CONSTRUCTED, true); if (!j->get_flag(JOB_FOREGROUND)) { proc_last_bg_pid = j->pgid; diff --git a/src/fish.cpp b/src/fish.cpp index 096805a6e..273c85ab8 100644 --- a/src/fish.cpp +++ b/src/fish.cpp @@ -387,6 +387,13 @@ int main(int argc, char **argv) { const io_chain_t empty_ios; if (read_init(paths)) { + // TODO: Remove this once we're confident that not blocking/unblocking every signal around + // some critical sections is no longer necessary. + env_var_t fish_no_signal_block = env_get_string(L"FISH_NO_SIGNAL_BLOCK"); + if (!fish_no_signal_block.missing_or_empty() && !from_string(fish_no_signal_block)) { + ignore_signal_block = false; + } + // Stomp the exit status of any initialization commands (issue #635). proc_set_last_status(STATUS_CMD_OK); diff --git a/src/history.cpp b/src/history.cpp index a293933b9..5b748c8ce 100644 --- a/src/history.cpp +++ b/src/history.cpp @@ -39,6 +39,7 @@ #include "parse_util.h" #include "path.h" #include "reader.h" +#include "signal.h" #include "wutil.h" // IWYU pragma: keep // Our history format is intended to be valid YAML. Here it is: @@ -997,6 +998,9 @@ bool history_t::load_old_if_needed(void) { if (loaded_old) return true; loaded_old = true; + // PCA not sure why signals were blocked here + // signal_block(); + bool ok = false; if (map_file(name, &mmap_start, &mmap_length, &mmap_file_id)) { // Here we've mapped the file. @@ -1005,6 +1009,7 @@ bool history_t::load_old_if_needed(void) { this->populate_from_mmap(); } + // signal_unblock(); return ok; } @@ -1404,6 +1409,8 @@ bool history_t::save_internal_via_appending() { return true; } + signal_block(); + // We are going to open the file, lock it, append to it, and then close it // After locking it, we need to stat the file at the path; if there is a new file there, it // means @@ -1491,6 +1498,8 @@ bool history_t::save_internal_via_appending() { close(history_fd); } + signal_unblock(); + // If someone has replaced the file, forget our file state. if (file_changed) { this->clear_file_state(); diff --git a/src/postfork.cpp b/src/postfork.cpp index 8b79c4f7f..4688cff73 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -277,6 +277,8 @@ int setup_child_process(process_t *p, const io_chain_t &io_chain) { signal_reset_handlers(); } + signal_unblock(); // remove all signal blocks + return ok ? 0 : -1; } diff --git a/src/proc.cpp b/src/proc.cpp index 07e00b618..97098cf81 100644 --- a/src/proc.cpp +++ b/src/proc.cpp @@ -789,7 +789,7 @@ bool terminal_give_to_job(job_t *j, int cont) { return true; } - signal_block(); + signal_block(true); //It may not be safe to call tcsetpgrp if we've already done so, as at that point we are no longer //the controlling process group for the terminal and no longer have permission to set the process @@ -872,12 +872,12 @@ bool terminal_give_to_job(job_t *j, int cont) { if (errno == ENOTTY) redirect_tty_output(); debug(1, _(L"Could not send job %d ('%ls') to foreground"), j->job_id, j->command_wcstr()); wperror(L"tcsetattr"); - signal_unblock(); + signal_unblock(true); return false; } } - signal_unblock(); + signal_unblock(true); return true; } @@ -890,12 +890,12 @@ static bool terminal_return_from_job(job_t *j) { return true; } - signal_block(); + signal_block(true); if (tcsetpgrp(STDIN_FILENO, getpgrp()) == -1) { if (errno == ENOTTY) redirect_tty_output(); debug(1, _(L"Could not return shell to foreground")); wperror(L"tcsetpgrp"); - signal_unblock(); + signal_unblock(true); return false; } @@ -904,7 +904,7 @@ static bool terminal_return_from_job(job_t *j) { if (errno == EIO) redirect_tty_output(); debug(1, _(L"Could not return shell to foreground")); wperror(L"tcgetattr"); - signal_unblock(); + signal_unblock(true); return false; } @@ -922,7 +922,7 @@ static bool terminal_return_from_job(job_t *j) { } #endif - signal_unblock(); + signal_unblock(true); return true; } diff --git a/src/reader.cpp b/src/reader.cpp index 7daccce2a..c7d3d8afd 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -1575,12 +1575,20 @@ static void reader_interactive_init() { // Check if we are in control of the terminal, so that we don't do semi-expensive things like // reset signal handlers unless we really have to, which we often don't. if (tcgetpgrp(STDIN_FILENO) != shell_pgid) { + int block_count = 0; + int i; + // Bummer, we are not in control of the terminal. Stop until parent has given us control of - // it. + // it. Stopping in fish is a bit of a challange, what with all the signal fidgeting, we need + // to reset a bunch of signal state, making this coda a but unobvious. // // In theory, reseting signal handlers could cause us to miss signal deliveries. In - // practice, this code should only be run during startup, when we're not waiting for any + // practice, this code should only be run suring startup, when we're not waiting for any // signals. + while (signal_is_blocked()) { + signal_unblock(); + block_count++; + } signal_reset_handlers(); // Ok, signal handlers are taken out of the picture. Stop ourself in a loop until we are in @@ -1624,6 +1632,10 @@ static void reader_interactive_init() { } signal_set_handlers(); + + for (i = 0; i < block_count; i++) { + signal_block(); + } } // Put ourselves in our own process group. diff --git a/src/signal.cpp b/src/signal.cpp index f36847b59..1b937825c 100644 --- a/src/signal.cpp +++ b/src/signal.cpp @@ -16,6 +16,9 @@ #include "reader.h" #include "wutil.h" // IWYU pragma: keep +// This is a temporary var while we explore whether signal_block() and friends is needed. +bool ignore_signal_block = true; + /// Struct describing an entry for the lookup table used to convert between signal names and signal /// ids, etc. struct lookup_entry { @@ -380,7 +383,9 @@ void get_signals_with_handlers(sigset_t *set) { } } -void signal_block() { +void signal_block(bool force) { + if (!force && ignore_signal_block) return; + ASSERT_IS_MAIN_THREAD(); sigset_t chldset; @@ -400,10 +405,14 @@ void signal_unblock_all() { sigprocmask(SIG_SETMASK, &iset, NULL); } -void signal_unblock() { +void signal_unblock(bool force) { + if (!force && ignore_signal_block) return; + ASSERT_IS_MAIN_THREAD(); + sigset_t chldset; block_count--; + if (block_count < 0) { debug(0, _(L"Signal block mismatch")); bugreport(); @@ -411,12 +420,13 @@ void signal_unblock() { } if (!block_count) { - sigset_t chldset; sigfillset(&chldset); DIE_ON_FAILURE(pthread_sigmask(SIG_UNBLOCK, &chldset, 0)); } + // debug( 0, L"signal block level decreased to %d", block_count ); } bool signal_is_blocked() { + if (ignore_signal_block) return false; return static_cast(block_count); } diff --git a/src/signal.h b/src/signal.h index dcbc3be75..06fe37cf9 100644 --- a/src/signal.h +++ b/src/signal.h @@ -30,14 +30,16 @@ void signal_handle(int sig, int do_handle); void signal_unblock_all(); /// Block all signals. -void signal_block(); +void signal_block(bool force = false); /// Unblock all signals. -void signal_unblock(); +void signal_unblock(bool force = false); /// Returns true if signals are being blocked. bool signal_is_blocked(); /// Returns signals with non-default handlers. void get_signals_with_handlers(sigset_t *set); + +extern bool ignore_signal_block; #endif From 82ee3d6a4e174c065c2c0dba0037ce30646e365f Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:24:08 -0700 Subject: [PATCH 34/79] Revert "Deduplication between INTERNAL_FUNCTION and INTERNAL_BLOCK_NODE" This reverts commit 0594735714faf03e92863ebda67945a45521493e. It was meant for the major branch. --- src/exec.cpp | 52 +++++++++++++++++++++++++++------------------------- 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 891159005..72e615c30 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -685,28 +685,6 @@ void exec_job(parser_t &parser, job_t *j) { return true; }; - - //helper routine executed by INTERNAL_FUNCTION and INTERNAL_BLOCK_NODE to make sure - //an output buffer exists in case there is another command in the job chain that will - //be reading from this command's output. - auto verify_buffer_output = [&] () { - if (!p->is_last_in_job) { - // Be careful to handle failure, e.g. too many open fds. - block_output_io_buffer = io_buffer_t::create(STDOUT_FILENO, all_ios); - if (block_output_io_buffer.get() == NULL) { - exec_error = true; - job_mark_process_as_failed(j, p); - } else { - // This looks sketchy, because we're adding this io buffer locally - they - // aren't in the process or job redirection list. Therefore select_try won't - // be able to read them. However we call block_output_io_buffer->read() - // below, which reads until EOF. So there's no need to select on this. - process_net_io_chain.push_back(block_output_io_buffer); - } - } - }; - - switch (p->type) { case INTERNAL_FUNCTION: { // Calls to function_get_definition might need to source a file as a part of @@ -737,7 +715,20 @@ void exec_job(parser_t &parser, job_t *j) { parser.forbid_function(func_name); - verify_buffer_output(); + if (!p->is_last_in_job) { + // Be careful to handle failure, e.g. too many open fds. + block_output_io_buffer = io_buffer_t::create(STDOUT_FILENO, all_ios); + if (block_output_io_buffer.get() == NULL) { + exec_error = true; + job_mark_process_as_failed(j, p); + } else { + // This looks sketchy, because we're adding this io buffer locally - they + // aren't in the process or job redirection list. Therefore select_try won't + // be able to read them. However we call block_output_io_buffer->read() + // below, which reads until EOF. So there's no need to select on this. + process_net_io_chain.push_back(block_output_io_buffer); + } + } if (!exec_error) { internal_exec_helper(parser, def, NODE_OFFSET_INVALID, TOP, @@ -751,7 +742,18 @@ void exec_job(parser_t &parser, job_t *j) { } case INTERNAL_BLOCK_NODE: { - verify_buffer_output(); + if (!p->is_last_in_job) { + block_output_io_buffer = io_buffer_t::create(STDOUT_FILENO, all_ios); + if (block_output_io_buffer.get() == NULL) { + // We failed (e.g. no more fds could be created). + exec_error = true; + job_mark_process_as_failed(j, p); + } else { + // See the comment above about it's OK to add an IO redirection to this + // local buffer, even though it won't be handled in select_try. + process_net_io_chain.push_back(block_output_io_buffer); + } + } if (!exec_error) { internal_exec_helper(parser, wcstring(), p->internal_block_node, TOP, @@ -860,7 +862,7 @@ void exec_job(parser_t &parser, job_t *j) { const int fg = j->get_flag(JOB_FOREGROUND); j->set_flag(JOB_FOREGROUND, false); - //main loop may need to perform a blocking read from previous command's output. + //main loop is performing a blocking read from previous command's output //make sure read source is not blocked unblock_previous(); From 5aa281942f59c424fe9ae0c88565589386158519 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:24:18 -0700 Subject: [PATCH 35/79] Revert "Corrected job_type for external command in debug log" This reverts commit 4dfb334db813c35950b11a0201953d32b4b4a858. It was meant for the major branch. --- src/exec.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/exec.cpp b/src/exec.cpp index 72e615c30..931277c3d 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -1113,7 +1113,7 @@ void exec_job(parser_t &parser, job_t *j) { } else #endif { - if (!do_fork(false, "external command", [&] { + if (!do_fork(false, "internal builtin", [&] { safe_launch_process(p, actual_cmd, argv, envv); })) { break; From 03a66e31f43ee1dbeb0fe68bceee3b66295e61aa Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:24:29 -0700 Subject: [PATCH 36/79] Revert "Unified all child/parent forking code in exec_job" This reverts commit 384879704ac5164ac6ab659bc668b8933e073b5b. It was meant for the major branch. --- src/exec.cpp | 136 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 81 insertions(+), 55 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 931277c3d..5ff4396db 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -629,8 +629,6 @@ void exec_job(parser_t &parser, job_t *j) { // a pipeline. shared_ptr block_output_io_buffer; - //used to SIGCONT the previously SIGSTOP'd process so the main loop or next command in job can read - //from its output. auto unblock_previous = [&j] () { //we've already called waitpid after forking the child, so we've guaranteed that they're //already SIGSTOP'd. Otherwise we'd be risking a deadlock because we can call SIGCONT before @@ -645,46 +643,6 @@ void exec_job(parser_t &parser, job_t *j) { // This is the io_streams we pass to internal builtins. std::unique_ptr builtin_io_streams(new io_streams_t(stdout_read_limit)); - auto do_fork = [&j, &p, &pid, &exec_error, &process_net_io_chain, &block_child, &child_forked] - (bool drain_threads, const char *fork_type, std::function child_action) -> bool { - pid = execute_fork(drain_threads); - if (pid == 0) { - // This is the child process. Setup redirections, print correct output to - // stdout and stderr, and then exit. - p->pid = getpid(); - blocked_pid = -1; - child_set_group(j, p); - // Make child processes pause after executing setup_child_process() to give down-chain - // commands in the job a chance to join their process group and read their pipes. - // The process will be resumed when the next command in the chain is started. - if (block_child) { - kill(p->pid, SIGSTOP); - } - //the parent will wake us up when we're ready to execute - setup_child_process(p, process_net_io_chain); - child_action(); - DIE("Child process returned control to do_fork lambda!"); - } - else { - if (pid < 0) { - debug(1, L"Failed to fork %s!\n", fork_type); - job_mark_process_as_failed(j, p); - exec_error = true; - return false; - } - // This is the parent process. Store away information on the child, and - // possibly give it control over the terminal. - debug(2, L"Fork #%d, pid %d: %s for '%ls'", g_fork_count, pid, fork_type, p->argv0()); - child_forked = true; - if (block_child) { - debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); - } - p->pid = pid; - } - - return true; - }; - switch (p->type) { case INTERNAL_FUNCTION: { // Calls to function_get_definition might need to source a file as a part of @@ -927,11 +885,33 @@ void exec_job(parser_t &parser, job_t *j) { if (block_output_io_buffer->out_buffer_size() > 0) { // We don't have to drain threads here because our child process is simple. - if (!do_fork(false, "internal block or function", [&] { - exec_write_and_exit(block_output_io_buffer->fd, buffer, count, status); - })) { - break; + pid = execute_fork(false); + if (pid == 0) { + // This is the child process. Write out the contents of the pipeline. + p->pid = getpid(); + blocked_pid = -1; + child_set_group(j, p); + // Make child processes pause after executing setup_child_process() to give down-chain + // commands in the job a chance to join their process group and read their pipes. + // The process will be resumed when the next command in the chain is started. + if (block_child) { + kill(p->pid, SIGSTOP); + } + setup_child_process(p, process_net_io_chain); + + exec_write_and_exit(block_output_io_buffer->fd, buffer, count, status); + } else { + // This is the parent process. Store away information on the child, and + // possibly give it control over the terminal. + debug(2, L"Fork #%d, pid %d: internal block or function for '%ls'", + g_fork_count, pid, p->argv0()); + child_forked = true; + if (block_child) { + debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); + } + p->pid = pid; } + } else { if (p->is_last_in_job) { proc_set_last_status(j->get_flag(JOB_NEGATE) ? (!status) : status); @@ -1037,11 +1017,33 @@ void exec_job(parser_t &parser, job_t *j) { fflush(stdout); fflush(stderr); - if (!do_fork(false, "internal builtin", [&] { - do_builtin_io(outbuff, outbuff_len, errbuff, errbuff_len); - exit_without_destructors(p->status); - })) { - break; + pid = execute_fork(false); + if (pid == 0) { + // This is the child process. Setup redirections, print correct output to + // stdout and stderr, and then exit. + p->pid = getpid(); + blocked_pid = -1; + child_set_group(j, p); + // Make child processes pause after executing setup_child_process() to give down-chain + // commands in the job a chance to join their process group and read their pipes. + // The process will be resumed when the next command in the chain is started. + if (block_child) { + kill(p->pid, SIGSTOP); + } + setup_child_process(p, process_net_io_chain); + + do_builtin_io(outbuff, outbuff_len, errbuff, errbuff_len); + exit_without_destructors(p->status); + } else { + // This is the parent process. Store away information on the child, and + // possibly give it control over the terminal. + debug(2, L"Fork #%d, pid %d: internal builtin for '%ls'", g_fork_count, pid, + p->argv0()); + child_forked = true; + if (block_child) { + debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); + } + p->pid = pid; } } @@ -1113,10 +1115,34 @@ void exec_job(parser_t &parser, job_t *j) { } else #endif { - if (!do_fork(false, "internal builtin", [&] { - safe_launch_process(p, actual_cmd, argv, envv); - })) { - break; + pid = execute_fork(false); + if (pid == 0) { + // This is the child process. + p->pid = getpid(); + blocked_pid = -1; + child_set_group(j, p); + // Make child processes pause after executing setup_child_process() to give down-chain + // commands in the job a chance to join their process group and read their pipes. + // The process will be resumed when the next command in the chain is started. + if (block_child) { + kill(p->pid, SIGSTOP); + } + setup_child_process(p, process_net_io_chain); + + safe_launch_process(p, actual_cmd, argv, envv); + // safe_launch_process _never_ returns... + DIE("safe_launch_process should not have returned"); + } else { + debug(2, L"Fork #%d, pid %d: external command '%s' from '%ls'", + g_fork_count, pid, p->argv0(), file ? file : L""); + if (pid < 0) { + job_mark_process_as_failed(j, p); + exec_error = true; + } + child_forked = true; + if (block_child) { + debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); + } } } From 6fe0cb194109a56b384f8e2539e8b0f189017c6f Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:24:36 -0700 Subject: [PATCH 37/79] Revert "Split internal_exec to its own function" This reverts commit 4a1de248bc5ea83c63904dc962467b1e220c1eb9. It was meant for the major branch. --- src/exec.cpp | 63 ++++++++++++++++++++++++---------------------------- 1 file changed, 29 insertions(+), 34 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 5ff4396db..8804c3e28 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -366,39 +366,6 @@ static bool can_use_posix_spawn_for_job(const job_t *job, const process_t *proce return result; } -void internal_exec(job_t *j, const io_chain_t &&all_ios) { - // Do a regular launch - but without forking first... - signal_block(); - - // setup_child_process makes sure signals are properly set up. It will also call - // signal_unblock. - - // PCA This is for handling exec. Passing all_ios here matches what fish 2.0.0 and 1.x did. - // It's known to be wrong - for example, it means that redirections bound for subsequent - // commands in the pipeline will apply to exec. However, using exec in a pipeline doesn't - // really make sense, so I'm not trying to fix it here. - if (!setup_child_process(0, all_ios)) { - // Decrement SHLVL as we're removing ourselves from the shell "stack". - const env_var_t shlvl_str = env_get_string(L"SHLVL", ENV_GLOBAL | ENV_EXPORT); - wcstring nshlvl_str = L"0"; - if (!shlvl_str.missing()) { - long shlvl_i = fish_wcstol(shlvl_str.c_str()); - if (!errno && shlvl_i > 0) { - nshlvl_str = to_string(shlvl_i - 1); - } - } - env_set(L"SHLVL", nshlvl_str.c_str(), ENV_GLOBAL | ENV_EXPORT); - - // launch_process _never_ returns. - launch_process_nofork(j->processes.front().get()); - } else { - j->set_flag(JOB_CONSTRUCTED, true); - j->processes.front()->completed = 1; - return; - } -} - - void exec_job(parser_t &parser, job_t *j) { pid_t pid = 0; @@ -434,7 +401,35 @@ void exec_job(parser_t &parser, job_t *j) { } if (j->processes.front()->type == INTERNAL_EXEC) { - internal_exec(j, std::move(all_ios)); + // Do a regular launch - but without forking first... + signal_block(); + + // setup_child_process makes sure signals are properly set up. It will also call + // signal_unblock. + + // PCA This is for handling exec. Passing all_ios here matches what fish 2.0.0 and 1.x did. + // It's known to be wrong - for example, it means that redirections bound for subsequent + // commands in the pipeline will apply to exec. However, using exec in a pipeline doesn't + // really make sense, so I'm not trying to fix it here. + if (!setup_child_process(0, all_ios)) { + // Decrement SHLVL as we're removing ourselves from the shell "stack". + const env_var_t shlvl_str = env_get(L"SHLVL", ENV_GLOBAL | ENV_EXPORT); + wcstring nshlvl_str = L"0"; + if (!shlvl_str.missing()) { + long shlvl_i = fish_wcstol(shlvl_str.c_str()); + if (!errno && shlvl_i > 0) { + nshlvl_str = to_string(shlvl_i - 1); + } + } + env_set(L"SHLVL", nshlvl_str.c_str(), ENV_GLOBAL | ENV_EXPORT); + + // launch_process _never_ returns. + launch_process_nofork(j->processes.front().get()); + } else { + j->set_flag(JOB_CONSTRUCTED, true); + j->processes.front()->completed = 1; + return; + } DIE("this should be unreachable"); } From 5c0311653e5ec2ad69233862a9df7f63111c8688 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:24:44 -0700 Subject: [PATCH 38/79] Revert "Raised debug level for "Retrying setpgid" message" This reverts commit 711c81b8c8bf5d2dcfffe5add8ca5ab39382bf8a. It was meant for the major branch. --- src/postfork.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/postfork.cpp b/src/postfork.cpp index 4688cff73..b0afec96b 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -83,7 +83,7 @@ bool child_set_group(job_t *j, process_t *p) { //times to get the kernel to see the new group. (Linux 4.4.0) int failure = setpgid(p->pid, j->pgid); while (failure == -1 && (errno == EPERM || errno == EINTR)) { - debug_safe(4, "Retrying setpgid in child process"); + debug_safe(1, "Retrying setpgid in child process"); failure = setpgid(p->pid, j->pgid); } // TODO: Figure out why we're testing whether the pgid is correct after attempting to From aab3dbd24ca50b11207d3391adaa121877da1927 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:25:02 -0700 Subject: [PATCH 39/79] Revert "Removed unused header include" This reverts commit 87db424e45a7b04ca45d9dca735b2368a123977f. It was meant for the major branch. --- src/exec.cpp | 1 + 1 file changed, 1 insertion(+) diff --git a/src/exec.cpp b/src/exec.cpp index 8804c3e28..918d98fa3 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include From 260d5bb013a727823625825e2c4df568a7450458 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:25:10 -0700 Subject: [PATCH 40/79] Revert "Cleaned up terminal_give_to_job() code flow and comments" This reverts commit 7e23965250eef92452df1ddd89f316a411f92a28. It was meant for the major branch. --- src/proc.cpp | 46 ++++++++++++++++++++++++---------------------- 1 file changed, 24 insertions(+), 22 deletions(-) diff --git a/src/proc.cpp b/src/proc.cpp index 97098cf81..a4b577b68 100644 --- a/src/proc.cpp +++ b/src/proc.cpp @@ -791,7 +791,10 @@ bool terminal_give_to_job(job_t *j, int cont) { signal_block(true); - //It may not be safe to call tcsetpgrp if we've already done so, as at that point we are no longer + //Previously, terminal_give_to_job was being called for each process in a job, hence all the comments + //and warnings below. It is now only called for the first process in a job.t d + + //it may not be safe to call tcsetpgrp if we've already done so, as at that point we are no longer //the controlling process group for the terminal and no longer have permission to set the process //group that is in control, causing tcsetpgrp to return EPERM, even though that's not the documented //behavior in tcsetpgrp(3), which instead says other bad things will happen (it says SIGTTOU will be @@ -804,7 +807,8 @@ bool terminal_give_to_job(job_t *j, int cont) { debug(2, L"Process group %d already has control of terminal\n", j->pgid); } else { - debug(4, L"Attempting to bring process group to foreground via tcsetpgrp for job->pgid %d\n", j->pgid); + debug(4, L"Attempting bring process group to foreground via tcsetpgrp for job->pgid %d\n", j->pgid); + debug(4, L"caller session id: %d, pgid %d has session id: %d\n", getsid(0), j->pgid, getsid(j->pgid)); //the tcsetpgrp(2) man page says that EPERM is thrown if "pgrp has a supported value, but is not the //process group ID of a process in the same session as the calling process." @@ -815,15 +819,22 @@ bool terminal_give_to_job(job_t *j, int cont) { //thing is that we can guarantee the process isn't going to exit while we wait (which would cause us to //possibly block indefinitely). + auto pgroupTerminated = [&j]() { + //everyone in the process group has exited + //The only way that can happen is if the very last process in the group terminated, and didn't need + //to access the terminal, otherwise it would have hung waiting for terminal IO. We can ignore this. + debug(3, L"terminal_give_to_job(): tcsetpgrp called but process group %d has terminated.\n", j->pgid); + }; + while (tcsetpgrp(STDIN_FILENO, j->pgid) != 0) { - bool pgroup_terminated = false; if (errno == EINTR) { - //always retry on EINTR, see comments in tcsetattr EINTR code below. + //always retry on EINTR } else if (errno == EINVAL) { //OS X returns EINVAL if the process group no longer lives. Probably other OSes, too. - //Unlike EPERM below, EINVAL can only happen if the process group has terminated - pgroup_terminated = true; + //See comments in pgroupTerminated() above. + pgroupTerminated(); + break; } else if (errno == EPERM) { //retry so long as this isn't because the process group is dead @@ -833,29 +844,19 @@ bool terminal_give_to_job(job_t *j, int cont) { //because no such process group exists any longer. This is the observed behavior on Linux 4.4.0. //a "success" result would mean processes from the group still exist but is still running in some state //or the other. - pgroup_terminated = true; - } - else { - //debug the original tcsetpgrp error (not the waitpid errno) to the log, and then retry until not EPERM - //or the process group has exited. - debug(2, L"terminal_give_to_job(): EPERM.\n", j->pgid); + //See comments in pgroupTerminated() above. + pgroupTerminated(); + break; } + debug(2, L"terminal_give_to_job(): EPERM.\n", j->pgid); } else { if (errno == ENOTTY) redirect_tty_output(); - debug(1, _(L"Could not send job %d ('%ls') with pgid %d to foreground"), j->job_id, j->command_wcstr(), j->pgid); + debug(1, _(L"terminal_give_to_job(): Could not send job %d ('%ls') with pgid %d to foreground"), j->job_id, j->command_wcstr(), j->pgid); wperror(L"tcsetpgrp"); signal_unblock(true); return false; } - - if (pgroup_terminated) { - //all processes in the process group has exited. Since we force all child procs to SIGSTOP on startup, - //the only way that can happen is if the very last process in the group terminated, and didn't need - //to access the terminal, otherwise it would have hung waiting for terminal IO (SIGTTIN). We can ignore this. - debug(3, L"tcsetpgrp called but process group %d has terminated.\n", j->pgid); - break; - } } } @@ -870,7 +871,8 @@ bool terminal_give_to_job(job_t *j, int cont) { } if (result == -1) { if (errno == ENOTTY) redirect_tty_output(); - debug(1, _(L"Could not send job %d ('%ls') to foreground"), j->job_id, j->command_wcstr()); + debug(1, _(L"terminal_give_to_job(): Could not send job %d ('%ls') to foreground"), + j->job_id, j->command_wcstr()); wperror(L"tcsetattr"); signal_unblock(true); return false; From 1bbd288f8649478321f4356073bfae68f3518328 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:25:19 -0700 Subject: [PATCH 41/79] Revert "Removed unused job_t * parameter from setup_child_process" This reverts commit dabe718c52f498f43e6dfc5c0e292cae2f7fd2b2. It was meant for the major branch. --- src/exec.cpp | 8 ++++---- src/postfork.cpp | 2 +- src/postfork.h | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 918d98fa3..89d24f06a 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -412,7 +412,7 @@ void exec_job(parser_t &parser, job_t *j) { // It's known to be wrong - for example, it means that redirections bound for subsequent // commands in the pipeline will apply to exec. However, using exec in a pipeline doesn't // really make sense, so I'm not trying to fix it here. - if (!setup_child_process(0, all_ios)) { + if (!setup_child_process(j, 0, all_ios)) { // Decrement SHLVL as we're removing ourselves from the shell "stack". const env_var_t shlvl_str = env_get(L"SHLVL", ENV_GLOBAL | ENV_EXPORT); wcstring nshlvl_str = L"0"; @@ -893,7 +893,7 @@ void exec_job(parser_t &parser, job_t *j) { if (block_child) { kill(p->pid, SIGSTOP); } - setup_child_process(p, process_net_io_chain); + setup_child_process(j, p, process_net_io_chain); exec_write_and_exit(block_output_io_buffer->fd, buffer, count, status); } else { @@ -1026,7 +1026,7 @@ void exec_job(parser_t &parser, job_t *j) { if (block_child) { kill(p->pid, SIGSTOP); } - setup_child_process(p, process_net_io_chain); + setup_child_process(j, p, process_net_io_chain); do_builtin_io(outbuff, outbuff_len, errbuff, errbuff_len); exit_without_destructors(p->status); @@ -1123,7 +1123,7 @@ void exec_job(parser_t &parser, job_t *j) { if (block_child) { kill(p->pid, SIGSTOP); } - setup_child_process(p, process_net_io_chain); + setup_child_process(j, p, process_net_io_chain); safe_launch_process(p, actual_cmd, argv, envv); // safe_launch_process _never_ returns... diff --git a/src/postfork.cpp b/src/postfork.cpp index b0afec96b..2973942ac 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -260,7 +260,7 @@ static int handle_child_io(const io_chain_t &io_chain) { return 0; } -int setup_child_process(process_t *p, const io_chain_t &io_chain) { +int setup_child_process(job_t *j, process_t *p, const io_chain_t &io_chain) { bool ok = true; if (ok) { diff --git a/src/postfork.h b/src/postfork.h index 4a3b47e4e..a90fd377b 100644 --- a/src/postfork.h +++ b/src/postfork.h @@ -33,7 +33,7 @@ bool child_set_group(job_t *j, process_t *p); //called by child /// /// \return 0 on sucess, -1 on failiure. When this function returns, signals are always unblocked. /// On failiure, signal handlers, io redirections and process group of the process is undefined. -int setup_child_process(process_t *p, const io_chain_t &io_chain); +int setup_child_process(job_t *j, process_t *p, const io_chain_t &io_chain); /// Call fork(), optionally waiting until we are no longer multithreaded. If the forked child /// doesn't do anything that could allocate memory, take a lock, etc. (like call exec), then it's From a3863b22a76b6cb99e48252b72fa47a7c6b99e4e Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:25:29 -0700 Subject: [PATCH 42/79] Revert "OS X EINVAL compatibility for waitpid" This reverts commit 628db65504a69cff1d04a0639ace4a9358504fff. It was meant for the major branch. --- src/exec.cpp | 9 +-------- src/proc.cpp | 23 ++++++----------------- 2 files changed, 7 insertions(+), 25 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 89d24f06a..23e604f48 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -1163,14 +1163,7 @@ void exec_job(parser_t &parser, job_t *j) { //but we only need to call set_child_group for the first process in the group. //If needs_keepalive is set, this has already been called for the keepalive process pid_t pid_status{}; - int result; - while ((result = waitpid(p->pid, &pid_status, WUNTRACED)) == -1 && errno == EINTR) { - //This could be a superfluous interrupt or Ctrl+C at the terminal - //In all cases, it is OK to retry since the forking code above is specifically designed - //to never, ever hang/block in a child process before the SIGSTOP call is reached. - continue; - } - if (result == -1) { + if (waitpid(p->pid, &pid_status, WUNTRACED) == -1) { exec_error = true; debug(1, L"waitpid(%d) call in unblock_pid failed:!\n", p->pid); wperror(L"waitpid"); diff --git a/src/proc.cpp b/src/proc.cpp index a4b577b68..37c267850 100644 --- a/src/proc.cpp +++ b/src/proc.cpp @@ -819,33 +819,22 @@ bool terminal_give_to_job(job_t *j, int cont) { //thing is that we can guarantee the process isn't going to exit while we wait (which would cause us to //possibly block indefinitely). - auto pgroupTerminated = [&j]() { - //everyone in the process group has exited - //The only way that can happen is if the very last process in the group terminated, and didn't need - //to access the terminal, otherwise it would have hung waiting for terminal IO. We can ignore this. - debug(3, L"terminal_give_to_job(): tcsetpgrp called but process group %d has terminated.\n", j->pgid); - }; - while (tcsetpgrp(STDIN_FILENO, j->pgid) != 0) { if (errno == EINTR) { //always retry on EINTR } - else if (errno == EINVAL) { - //OS X returns EINVAL if the process group no longer lives. Probably other OSes, too. - //See comments in pgroupTerminated() above. - pgroupTerminated(); - break; - } else if (errno == EPERM) { - //retry so long as this isn't because the process group is dead + //so long as this isn't because the process group is dead int wait_result = waitpid(-1 * j->pgid, &wait_result, WNOHANG); if (wait_result == -1) { + //everyone in the process group has exited + //The only way that can happen is if the very last process in the group terminated, and didn't need + //to access the terminal, otherwise it would have hung waiting for terminal IO. We can ignore this. //Note that -1 is technically an "error" for waitpid in the sense that an invalid argument was specified - //because no such process group exists any longer. This is the observed behavior on Linux 4.4.0. + //because no such process group exists any longer. //a "success" result would mean processes from the group still exist but is still running in some state //or the other. - //See comments in pgroupTerminated() above. - pgroupTerminated(); + debug(3, L"terminal_give_to_job(): tcsetpgrp called but process group %d has terminated.\n", j->pgid); break; } debug(2, L"terminal_give_to_job(): EPERM.\n", j->pgid); From 177256064c5d8eaaa31f3e4a5b152a79cd46389b Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:25:45 -0700 Subject: [PATCH 43/79] Revert "Removed old/unneeded variants of block_child" This reverts commit 8e63386203dcf6377a9c5a49207249523d66ccd3. It was meant for the major branch. --- src/exec.cpp | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/exec.cpp b/src/exec.cpp index 23e604f48..1251e5ed0 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -519,6 +519,8 @@ void exec_job(parser_t &parser, job_t *j) { //set to true if we end up forking for this process bool child_forked = false; bool child_spawned = false; + // bool block_child = !needs_keepalive; + // bool block_child = pipes_to_next_command; bool block_child = true; // The pipes the current process write to and read from. Unfortunately these can't be just From a4593d011a5e1233f78819cce7c777db1019c85c Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:25:54 -0700 Subject: [PATCH 44/79] Revert "Added important comment about blocked_pid" This reverts commit 16d2f4faff4d06c26a162c50fb47fc2f1f0f73e0. It was meant for the major branch. --- src/exec.cpp | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 1251e5ed0..97669ba6f 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -496,8 +496,6 @@ void exec_job(parser_t &parser, job_t *j) { // We are careful to set these to -1 when closed, so if we exit the loop abruptly, we can still // close them. bool pgrp_set = false; - //this is static since processes can block on input/output across jobs - //the main exec_job loop is only ever run in a single thread, so this is OK static pid_t blocked_pid = -1; int pipe_current_read = -1, pipe_current_write = -1, pipe_next_read = -1; for (std::unique_ptr &unique_p : j->processes) { From d0ce2b4824c9b8a00ef838e6a3a2519db584640f Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:26:03 -0700 Subject: [PATCH 45/79] Revert "Minor refactoring" This reverts commit 15da6f02032f8bfa6d2af333cf386250ca76d4f5. It was meant for the major branch. --- src/exec.cpp | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 97669ba6f..761705b80 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -632,8 +632,8 @@ void exec_job(parser_t &parser, job_t *j) { if (blocked_pid != -1) { debug(2, L"Unblocking process %d.\n", blocked_pid); kill(blocked_pid, SIGCONT); - blocked_pid = -1; } + blocked_pid = -1; }; // This is the io_streams we pass to internal builtins. @@ -1174,13 +1174,15 @@ void exec_job(parser_t &parser, job_t *j) { //no one awaits it. } //regardless of whether the child blocked or not: - //only once per job, and only once we've actually executed an external command - if ((child_spawned || child_forked) && !pgrp_set) { + if (child_spawned || child_forked) { //this should be called after waitpid if child_forked && pipes_to_next_command //it can be called at any time if child_spawned - set_child_group(j, p->pid); - //we can't rely on p->is_first_in_job because a builtin may have been the first - pgrp_set = true; + if (!pgrp_set) { + set_child_group(j, p->pid); + //only once per job, and only once we've executed an external command for real + //we can't rely on p->is_first_in_job because a builtin may have been the first + pgrp_set = true; + } } //if the command we ran _before_ this one was SIGSTOP'd to let this one catch up, unblock it now. //this must be after the wait_pid on the process we just started, if any. From 376cb99974cf48e4acecd4755314fb0871cfc40e Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:26:13 -0700 Subject: [PATCH 46/79] Revert "Logging updates" This reverts commit a0efae5f08f6b157a397444a2c60a3088c6f7057. It was meant for the major branch. --- src/postfork.cpp | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/src/postfork.cpp b/src/postfork.cpp index 2973942ac..cb1e29fd9 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -83,7 +83,7 @@ bool child_set_group(job_t *j, process_t *p) { //times to get the kernel to see the new group. (Linux 4.4.0) int failure = setpgid(p->pid, j->pgid); while (failure == -1 && (errno == EPERM || errno == EINTR)) { - debug_safe(1, "Retrying setpgid in child process"); + debug_safe(2, "Retrying setpgid in child process"); failure = setpgid(p->pid, j->pgid); } // TODO: Figure out why we're testing whether the pgid is correct after attempting to @@ -149,8 +149,7 @@ bool set_child_group(job_t *j, pid_t child_pid) { //handle it).. debug(4, L"Process group %d already has control of terminal\n", j->pgid); } - else - { + else { //no need to duplicate the code here, a function already exists that does just this retval = terminal_give_to_job(j, false /*new job, so not continuing*/); } @@ -267,7 +266,7 @@ int setup_child_process(job_t *j, process_t *p, const io_chain_t &io_chain) { //In the case of IO_FILE, this can hang until data is available to read/write! ok = (0 == handle_child_io(io_chain)); if (p != 0 && !ok) { - debug_safe(4, "handle_child_io failed in setup_child_process"); + debug_safe(0, "p!=0 && !ok"); exit_without_destructors(1); } } @@ -294,7 +293,6 @@ pid_t execute_fork(bool wait_for_threads_to_die) { // Make sure we have no outstanding threads before we fork. This is a pretty sketchy thing // to do here, both because exec.cpp shouldn't have to know about iothreads, and because the // completion handlers may do unexpected things. - debug_safe(4, "waiting for threads to drain."); iothread_drain_all(); } From 8c86d258e87942aeec612ad3b0cb872efc9a0cf9 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:26:22 -0700 Subject: [PATCH 47/79] Revert "unblock_previous on exec_job finish" This reverts commit 5db8065f151d01e9003c24d5615a083acb6b33f0. It was meant for the major branch. --- src/exec.cpp | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 761705b80..258133261 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -1205,11 +1205,6 @@ void exec_job(parser_t &parser, job_t *j) { exec_close(pipe_current_write); pipe_current_write = -1; } - - //unblock the last process because there's no need for it to stay SIGSTOP'd for anything - if (p->is_last_in_job) { - unblock_previous(); - } } // Clean up any file descriptors we left open. From bd601019fe7c90af585cee443fcca573165701ca Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:26:30 -0700 Subject: [PATCH 48/79] Revert "blocking only if pipes_to_next_command breaks things like read.expect test" This reverts commit c3d756b5df0a2fdbe91cf892325249ee255b57ed. It was meant for the major branch. --- src/exec.cpp | 1 - 1 file changed, 1 deletion(-) diff --git a/src/exec.cpp b/src/exec.cpp index 258133261..bed8bf27c 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -518,7 +518,6 @@ void exec_job(parser_t &parser, job_t *j) { bool child_forked = false; bool child_spawned = false; // bool block_child = !needs_keepalive; - // bool block_child = pipes_to_next_command; bool block_child = true; // The pipes the current process write to and read from. Unfortunately these can't be just From 7b443ac1d3af6c6e1fbd1a0f7f6bac4fb84ad883 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:26:41 -0700 Subject: [PATCH 49/79] Revert "terminal_give_to_job() was bypassing the `cont` branch" This reverts commit b27217e106313fd492be093999bd8c3bd01b7221. It was meant for the major branch. --- src/proc.cpp | 67 ++++++++++++++++++++-------------------------------- 1 file changed, 25 insertions(+), 42 deletions(-) diff --git a/src/proc.cpp b/src/proc.cpp index 37c267850..dc43108b3 100644 --- a/src/proc.cpp +++ b/src/proc.cpp @@ -789,8 +789,6 @@ bool terminal_give_to_job(job_t *j, int cont) { return true; } - signal_block(true); - //Previously, terminal_give_to_job was being called for each process in a job, hence all the comments //and warnings below. It is now only called for the first process in a job.t d @@ -803,50 +801,35 @@ bool terminal_give_to_job(job_t *j, int cont) { //for SIGTTOU are installed. Read: http://curiousthing.org/sigttin-sigttou-deep-dive-linux //In all cases, our goal here was just to hand over control of the terminal to this process group, //which is a no-op if it's already been done. - if (tcgetpgrp(STDIN_FILENO) == j->pgid) { + auto previous_owner = tcgetpgrp(STDIN_FILENO); + if (previous_owner == j->pgid) { debug(2, L"Process group %d already has control of terminal\n", j->pgid); + return true; } - else { - debug(4, L"Attempting bring process group to foreground via tcsetpgrp for job->pgid %d\n", j->pgid); - debug(4, L"caller session id: %d, pgid %d has session id: %d\n", getsid(0), j->pgid, getsid(j->pgid)); + debug(4, L"Attempting bring process group to foreground via tcsetpgrp for job->pgid %d\n", j->pgid); + debug(4, L"caller session id: %d, pgid %d has session id: %d\n", getsid(0), j->pgid, getsid(j->pgid)); - //the tcsetpgrp(2) man page says that EPERM is thrown if "pgrp has a supported value, but is not the - //process group ID of a process in the same session as the calling process." - //Since we _guarantee_ that this isn't the case (the child calls setpgid before it calls SIGSTOP, and - //the child was created in the same session as us), it seems that EPERM is being thrown because of an - //caching issue - the call to tcsetpgrp isn't seeing the newly-created process group just yet. On this - //developer's test machine (WSL running Linux 4.4.0), EPERM does indeed disappear on retry. The important - //thing is that we can guarantee the process isn't going to exit while we wait (which would cause us to - //possibly block indefinitely). + //the tcsetpgrp(2) man page says that EPERM is thrown if "pgrp has a supported value, but is not the + //process group ID of a process in the same session as the calling process." + //Since we _guarantee_ that this isn't the case (the child calls setpgid before it calls SIGSTOP, and + //the child was created in the same session as us), it seems that EPERM is being thrown because of an + //caching issue - the call to tcsetpgrp isn't seeing the newly-created process group just yet. On this + //developer's test machine (WSL running Linux 4.4.0), EPERM does indeed disappear on retry. The important + //thing is that we can guarantee the process isn't going to exit while we wait (which would cause us to + //possibly block indefinitely). - while (tcsetpgrp(STDIN_FILENO, j->pgid) != 0) { - if (errno == EINTR) { - //always retry on EINTR - } - else if (errno == EPERM) { - //so long as this isn't because the process group is dead - int wait_result = waitpid(-1 * j->pgid, &wait_result, WNOHANG); - if (wait_result == -1) { - //everyone in the process group has exited - //The only way that can happen is if the very last process in the group terminated, and didn't need - //to access the terminal, otherwise it would have hung waiting for terminal IO. We can ignore this. - //Note that -1 is technically an "error" for waitpid in the sense that an invalid argument was specified - //because no such process group exists any longer. - //a "success" result would mean processes from the group still exist but is still running in some state - //or the other. - debug(3, L"terminal_give_to_job(): tcsetpgrp called but process group %d has terminated.\n", j->pgid); - break; - } - debug(2, L"terminal_give_to_job(): EPERM.\n", j->pgid); - } - else { - if (errno == ENOTTY) redirect_tty_output(); - debug(1, _(L"terminal_give_to_job(): Could not send job %d ('%ls') with pgid %d to foreground"), j->job_id, j->command_wcstr(), j->pgid); - wperror(L"tcsetpgrp"); - signal_unblock(true); - return false; - } - } + signal_block(true); + int result = -1; + errno = EINTR; + while (result == -1 && (errno == EINTR || errno == EPERM)) { + result = tcsetpgrp(STDIN_FILENO, j->pgid); + } + if (result == -1) { + if (errno == ENOTTY) redirect_tty_output(); + debug(1, _(L"terminal_give_to_job(): Could not send job %d ('%ls') with pgid %d to foreground"), j->job_id, j->command_wcstr(), j->pgid); + wperror(L"tcsetpgrp"); + signal_unblock(true); + return false; } if (cont) { From 4b1bc53f9189741b9f4525a8ef5db99542aae4b8 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:26:52 -0700 Subject: [PATCH 50/79] Revert "Split child_set_group from setup_child_process" This reverts commit f7b051905e66abf3bbdd8c6f0f2ce75f2d0a73a2. It was meant for the major branch. --- src/exec.cpp | 29 +++++++++++------------------ src/postfork.cpp | 10 +++++----- 2 files changed, 16 insertions(+), 23 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index bed8bf27c..f0e1f6af8 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -517,8 +517,7 @@ void exec_job(parser_t &parser, job_t *j) { //set to true if we end up forking for this process bool child_forked = false; bool child_spawned = false; - // bool block_child = !needs_keepalive; - bool block_child = true; + // bool child_blocked = false; // The pipes the current process write to and read from. Unfortunately these can't be just // allocated on the stack, since j->io wants shared_ptr. @@ -884,15 +883,13 @@ void exec_job(parser_t &parser, job_t *j) { if (pid == 0) { // This is the child process. Write out the contents of the pipeline. p->pid = getpid(); - blocked_pid = -1; - child_set_group(j, p); + setup_child_process(j, p, process_net_io_chain); // Make child processes pause after executing setup_child_process() to give down-chain // commands in the job a chance to join their process group and read their pipes. // The process will be resumed when the next command in the chain is started. - if (block_child) { + if (pipes_to_next_command) { kill(p->pid, SIGSTOP); } - setup_child_process(j, p, process_net_io_chain); exec_write_and_exit(block_output_io_buffer->fd, buffer, count, status); } else { @@ -901,7 +898,7 @@ void exec_job(parser_t &parser, job_t *j) { debug(2, L"Fork #%d, pid %d: internal block or function for '%ls'", g_fork_count, pid, p->argv0()); child_forked = true; - if (block_child) { + if (pipes_to_next_command) { debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); } p->pid = pid; @@ -1017,15 +1014,13 @@ void exec_job(parser_t &parser, job_t *j) { // This is the child process. Setup redirections, print correct output to // stdout and stderr, and then exit. p->pid = getpid(); - blocked_pid = -1; - child_set_group(j, p); + setup_child_process(j, p, process_net_io_chain); // Make child processes pause after executing setup_child_process() to give down-chain // commands in the job a chance to join their process group and read their pipes. // The process will be resumed when the next command in the chain is started. - if (block_child) { + if (pipes_to_next_command) { kill(p->pid, SIGSTOP); } - setup_child_process(j, p, process_net_io_chain); do_builtin_io(outbuff, outbuff_len, errbuff, errbuff_len); exit_without_destructors(p->status); @@ -1035,7 +1030,7 @@ void exec_job(parser_t &parser, job_t *j) { debug(2, L"Fork #%d, pid %d: internal builtin for '%ls'", g_fork_count, pid, p->argv0()); child_forked = true; - if (block_child) { + if (pipes_to_next_command) { debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); } p->pid = pid; @@ -1114,15 +1109,13 @@ void exec_job(parser_t &parser, job_t *j) { if (pid == 0) { // This is the child process. p->pid = getpid(); - blocked_pid = -1; - child_set_group(j, p); + setup_child_process(j, p, process_net_io_chain); // Make child processes pause after executing setup_child_process() to give down-chain // commands in the job a chance to join their process group and read their pipes. // The process will be resumed when the next command in the chain is started. - if (block_child) { + if (pipes_to_next_command) { kill(p->pid, SIGSTOP); } - setup_child_process(j, p, process_net_io_chain); safe_launch_process(p, actual_cmd, argv, envv); // safe_launch_process _never_ returns... @@ -1135,7 +1128,7 @@ void exec_job(parser_t &parser, job_t *j) { exec_error = true; } child_forked = true; - if (block_child) { + if (pipes_to_next_command) { debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); } } @@ -1154,7 +1147,7 @@ void exec_job(parser_t &parser, job_t *j) { } } - bool child_blocked = block_child && child_forked; + bool child_blocked = child_forked && pipes_to_next_command; //to make things clearer below if (child_blocked) { //we have to wait to ensure the child has set their progress group and is in SIGSTOP state //otherwise, we can later call SIGCONT before they call SIGSTOP and they'll be blocked indefinitely. diff --git a/src/postfork.cpp b/src/postfork.cpp index cb1e29fd9..ecfd910fe 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -125,9 +125,6 @@ bool child_set_group(job_t *j, process_t *p) { /// and we can give the new process group control of the terminal if it's to run in the foreground. Note that /// we can guarantee the child won't try to read from the terminal before we've had a chance to run this code, /// because we haven't woken them up with a SIGCONT yet. -/// This musn't be called as a part of setup_child_process because that can hang indefinitely until data is -/// available to read/write in the case of IO_FILE, which means we'll never reach our SIGSTOP and everything -/// hangs. bool set_child_group(job_t *j, pid_t child_pid) { bool retval = true; @@ -262,11 +259,14 @@ static int handle_child_io(const io_chain_t &io_chain) { int setup_child_process(job_t *j, process_t *p, const io_chain_t &io_chain) { bool ok = true; + //p is zero when EXEC_INTERNAL + if (p) { + ok = child_set_group(j, p); + } + if (ok) { - //In the case of IO_FILE, this can hang until data is available to read/write! ok = (0 == handle_child_io(io_chain)); if (p != 0 && !ok) { - debug_safe(0, "p!=0 && !ok"); exit_without_destructors(1); } } From ac4eedb4362e3ff19a1cedc55df57d4dcf3835f9 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:27:00 -0700 Subject: [PATCH 51/79] Revert "Fixed no-op loop hang" This reverts commit 8537cc982e67d548b08dcc887ff32d338e7ba3e4. It was meant for the major branch. --- src/postfork.cpp | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/postfork.cpp b/src/postfork.cpp index ecfd910fe..1e6e9fd29 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -82,9 +82,8 @@ bool child_set_group(job_t *j, process_t *p) { //because we are SIGSTOPing the previous job in the chain. Sometimes we have to try a few //times to get the kernel to see the new group. (Linux 4.4.0) int failure = setpgid(p->pid, j->pgid); - while (failure == -1 && (errno == EPERM || errno == EINTR)) { + while (failure == -1 && errno == EPERM) { debug_safe(2, "Retrying setpgid in child process"); - failure = setpgid(p->pid, j->pgid); } // TODO: Figure out why we're testing whether the pgid is correct after attempting to // set it failed. This was added in commit 4e912ef8 from 2012-02-27. From 92628a384b89dc6a9756240e4e934e91d8273d5d Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:27:08 -0700 Subject: [PATCH 52/79] Revert "Retry setpgid in setup_child_process on EPERM" This reverts commit d6c4e66484957355d4985938ca30bc5d31ddc156. It was meant for the major branch. --- src/exec.cpp | 5 +++-- src/postfork.cpp | 6 ------ 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index f0e1f6af8..ec6d299c8 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -935,10 +935,11 @@ void exec_job(parser_t &parser, job_t *j) { // output, so that we can truncate the file. Does not apply to /dev/null. bool must_fork = redirection_is_to_real_file(stdout_io.get()) || redirection_is_to_real_file(stderr_io.get()); - if (!must_fork && p->is_last_in_job) { + if (!must_fork) { //we are handling reads directly in the main loop. Make sure source is unblocked. - //Note that we may still end up forking. unblock_previous(); + } + if (!must_fork && p->is_last_in_job) { const bool stdout_is_to_buffer = stdout_io && stdout_io->io_mode == IO_BUFFER; const bool no_stdout_output = stdout_buffer.empty(); const bool no_stderr_output = stderr_buffer.empty(); diff --git a/src/postfork.cpp b/src/postfork.cpp index 1e6e9fd29..999324bd1 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -78,13 +78,7 @@ bool child_set_group(job_t *j, process_t *p) { if (j->pgid == -2) { j->pgid = p->pid; } - //retry on EPERM because there's no way that a child cannot join an existing progress group - //because we are SIGSTOPing the previous job in the chain. Sometimes we have to try a few - //times to get the kernel to see the new group. (Linux 4.4.0) int failure = setpgid(p->pid, j->pgid); - while (failure == -1 && errno == EPERM) { - debug_safe(2, "Retrying setpgid in child process"); - } // TODO: Figure out why we're testing whether the pgid is correct after attempting to // set it failed. This was added in commit 4e912ef8 from 2012-02-27. failure = failure && getpgid(p->pid) != j->pgid; From b3f43723dcb5b596b78090a477940604eb1f127b Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:27:17 -0700 Subject: [PATCH 53/79] Revert "Improved comments" This reverts commit 1ae0272c4edee47390f7bc59d7c02fbcf21f0c37. It was meant for the major branch. --- src/exec.cpp | 1 - 1 file changed, 1 deletion(-) diff --git a/src/exec.cpp b/src/exec.cpp index ec6d299c8..dd5763128 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -1173,7 +1173,6 @@ void exec_job(parser_t &parser, job_t *j) { if (!pgrp_set) { set_child_group(j, p->pid); //only once per job, and only once we've executed an external command for real - //we can't rely on p->is_first_in_job because a builtin may have been the first pgrp_set = true; } } From 607d9e6aef0bd412c57420352a4ccaac85f38f4f Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:27:40 -0700 Subject: [PATCH 54/79] Revert "No need to unblock last process since it will no longer be SIGSTOP'd" This reverts commit 30aa8b3663008afb1275ab7c90a30073b4235639. It was meant for the major branch. --- src/exec.cpp | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/exec.cpp b/src/exec.cpp index dd5763128..9f05a5d8e 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -1199,6 +1199,13 @@ void exec_job(parser_t &parser, job_t *j) { } } + //unblock the last process in the group + if (blocked_pid != -1) + { + kill(blocked_pid, SIGCONT); + blocked_pid = -1; + } + // Clean up any file descriptors we left open. if (pipe_current_read >= 0) exec_close(pipe_current_read); if (pipe_current_write >= 0) exec_close(pipe_current_write); From 8a1af4a38d9e36b0856c54e5fda43de6de559aab Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:27:48 -0700 Subject: [PATCH 55/79] Revert "Be more judicious about when SIGSTOP is performed" This reverts commit abf6874a2df910453948a91ac3dc767e64aff6d8. It was meant for the major branch. --- src/exec.cpp | 75 +++++++++++++++++++++------------------------------- 1 file changed, 30 insertions(+), 45 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 9f05a5d8e..86bc7ced1 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -495,7 +495,6 @@ void exec_job(parser_t &parser, job_t *j) { // // We are careful to set these to -1 when closed, so if we exit the loop abruptly, we can still // close them. - bool pgrp_set = false; static pid_t blocked_pid = -1; int pipe_current_read = -1, pipe_current_write = -1, pipe_next_read = -1; for (std::unique_ptr &unique_p : j->processes) { @@ -517,7 +516,6 @@ void exec_job(parser_t &parser, job_t *j) { //set to true if we end up forking for this process bool child_forked = false; bool child_spawned = false; - // bool child_blocked = false; // The pipes the current process write to and read from. Unfortunately these can't be just // allocated on the stack, since j->io wants shared_ptr. @@ -639,6 +637,10 @@ void exec_job(parser_t &parser, job_t *j) { switch (p->type) { case INTERNAL_FUNCTION: { + //internal function never forks + //unblock previous (if any) to prevent hang + unblock_previous(); + // Calls to function_get_definition might need to source a file as a part of // autoloading, hence there must be no blocks. signal_unblock(); @@ -814,12 +816,11 @@ void exec_job(parser_t &parser, job_t *j) { const int fg = j->get_flag(JOB_FOREGROUND); j->set_flag(JOB_FOREGROUND, false); + signal_unblock(); + //main loop is performing a blocking read from previous command's output //make sure read source is not blocked unblock_previous(); - - signal_unblock(); - p->status = builtin_run(parser, p->get_argv(), *builtin_io_streams); signal_block(); @@ -887,9 +888,7 @@ void exec_job(parser_t &parser, job_t *j) { // Make child processes pause after executing setup_child_process() to give down-chain // commands in the job a chance to join their process group and read their pipes. // The process will be resumed when the next command in the chain is started. - if (pipes_to_next_command) { - kill(p->pid, SIGSTOP); - } + kill(p->pid, SIGSTOP); exec_write_and_exit(block_output_io_buffer->fd, buffer, count, status); } else { @@ -898,9 +897,7 @@ void exec_job(parser_t &parser, job_t *j) { debug(2, L"Fork #%d, pid %d: internal block or function for '%ls'", g_fork_count, pid, p->argv0()); child_forked = true; - if (pipes_to_next_command) { - debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); - } + debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); p->pid = pid; } @@ -1019,9 +1016,7 @@ void exec_job(parser_t &parser, job_t *j) { // Make child processes pause after executing setup_child_process() to give down-chain // commands in the job a chance to join their process group and read their pipes. // The process will be resumed when the next command in the chain is started. - if (pipes_to_next_command) { - kill(p->pid, SIGSTOP); - } + kill(p->pid, SIGSTOP); do_builtin_io(outbuff, outbuff_len, errbuff, errbuff_len); exit_without_destructors(p->status); @@ -1031,9 +1026,7 @@ void exec_job(parser_t &parser, job_t *j) { debug(2, L"Fork #%d, pid %d: internal builtin for '%ls'", g_fork_count, pid, p->argv0()); child_forked = true; - if (pipes_to_next_command) { - debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); - } + debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); p->pid = pid; } } @@ -1114,9 +1107,7 @@ void exec_job(parser_t &parser, job_t *j) { // Make child processes pause after executing setup_child_process() to give down-chain // commands in the job a chance to join their process group and read their pipes. // The process will be resumed when the next command in the chain is started. - if (pipes_to_next_command) { - kill(p->pid, SIGSTOP); - } + kill(p->pid, SIGSTOP); safe_launch_process(p, actual_cmd, argv, envv); // safe_launch_process _never_ returns... @@ -1129,9 +1120,7 @@ void exec_job(parser_t &parser, job_t *j) { exec_error = true; } child_forked = true; - if (pipes_to_next_command) { - debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); - } + debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); } } @@ -1148,39 +1137,35 @@ void exec_job(parser_t &parser, job_t *j) { } } - bool child_blocked = child_forked && pipes_to_next_command; //to make things clearer below - if (child_blocked) { + if (child_spawned) { + //posix_spawn'd processes won't SIGSTOP + set_child_group(j, p->pid); + } + else if (child_forked) { //we have to wait to ensure the child has set their progress group and is in SIGSTOP state //otherwise, we can later call SIGCONT before they call SIGSTOP and they'll be blocked indefinitely. - //The child is SIGSTOP'd and is guaranteed to have called child_set_group() at this point. - //but we only need to call set_child_group for the first process in the group. - //If needs_keepalive is set, this has already been called for the keepalive process pid_t pid_status{}; - if (waitpid(p->pid, &pid_status, WUNTRACED) == -1) { - exec_error = true; + if (waitpid(p->pid, &pid_status, WUNTRACED) != -1) { + //the child is SIGSTOP'd and is guaranteed to have called child_set_group() at this point. + //but we only need to call set_child_group for the first process in the group. + //If needs_keepalive is set, this has already been called for the keepalive process + if (p->is_first_in_job) { + set_child_group(j, p->pid); + } + //we are not unblocking the child via SIGCONT just yet to give the next process a chance to open + //the pipes and join the process group. We only unblock the last process in the job chain because + //no one awaits it. + } + else { debug(1, L"waitpid(%d) call in unblock_pid failed:!\n", p->pid); wperror(L"waitpid"); - break; - } - //we are not unblocking the child via SIGCONT just yet to give the next process a chance to open - //the pipes and join the process group. We only unblock the last process in the job chain because - //no one awaits it. - } - //regardless of whether the child blocked or not: - if (child_spawned || child_forked) { - //this should be called after waitpid if child_forked && pipes_to_next_command - //it can be called at any time if child_spawned - if (!pgrp_set) { - set_child_group(j, p->pid); - //only once per job, and only once we've executed an external command for real - pgrp_set = true; } } //if the command we ran _before_ this one was SIGSTOP'd to let this one catch up, unblock it now. //this must be after the wait_pid on the process we just started, if any. unblock_previous(); - if (child_blocked) { + if (child_forked) { //store the newly-blocked command's PID so that it can be SIGCONT'd once the next process //in the chain is started. That may be in this job or in another job. blocked_pid = p->pid; From a8a8d33fe0b0cd1981729b05643096f217295f5e Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:27:57 -0700 Subject: [PATCH 56/79] Revert "Better set_child_group logic for multi-process jobs" This reverts commit 99c6f65fee8be5476197cb11d9c8f8bec85266b5. It was meant for the major branch. --- src/exec.cpp | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 86bc7ced1..376818dd4 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -1138,7 +1138,7 @@ void exec_job(parser_t &parser, job_t *j) { } if (child_spawned) { - //posix_spawn'd processes won't SIGSTOP + child_set_group(j, p); set_child_group(j, p->pid); } else if (child_forked) { @@ -1147,17 +1147,17 @@ void exec_job(parser_t &parser, job_t *j) { pid_t pid_status{}; if (waitpid(p->pid, &pid_status, WUNTRACED) != -1) { //the child is SIGSTOP'd and is guaranteed to have called child_set_group() at this point. - //but we only need to call set_child_group for the first process in the group. - //If needs_keepalive is set, this has already been called for the keepalive process - if (p->is_first_in_job) { - set_child_group(j, p->pid); - } + set_child_group(j, p->pid); //update our own process group info to match //we are not unblocking the child via SIGCONT just yet to give the next process a chance to open //the pipes and join the process group. We only unblock the last process in the job chain because //no one awaits it. + if (!pipes_to_next_command) + { + kill(p->pid, SIGCONT); + } } else { - debug(1, L"waitpid(%d) call in unblock_pid failed:!\n", p->pid); + debug(2, L"waitpid(%d) call in unblock_pid failed:!\n", p->pid); wperror(L"waitpid"); } } @@ -1184,13 +1184,6 @@ void exec_job(parser_t &parser, job_t *j) { } } - //unblock the last process in the group - if (blocked_pid != -1) - { - kill(blocked_pid, SIGCONT); - blocked_pid = -1; - } - // Clean up any file descriptors we left open. if (pipe_current_read >= 0) exec_close(pipe_current_read); if (pipe_current_write >= 0) exec_close(pipe_current_write); From b023bddc8879c118c57f1ef809c287b6a99f22a1 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:28:05 -0700 Subject: [PATCH 57/79] Revert "Clarified job_continue logging" This reverts commit 8f2ef082be3679a76156f4fdcff459aec45ee435. It was meant for the major branch. --- src/proc.cpp | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/proc.cpp b/src/proc.cpp index dc43108b3..a6ee9e319 100644 --- a/src/proc.cpp +++ b/src/proc.cpp @@ -906,9 +906,7 @@ void job_continue(job_t *j, bool cont) { j->set_flag(JOB_NOTIFIED, false); CHECK_BLOCK(); - debug(4, L"%ls job %d, gid %d (%ls), %ls, %ls", - cont ? L"Continue" : L"Start", - j->job_id, j->pgid, j->command_wcstr(), + debug(4, L"Continue job %d, gid %d (%ls), %ls, %ls", j->job_id, j->pgid, j->command_wcstr(), job_is_completed(j) ? L"COMPLETED" : L"UNCOMPLETED", is_interactive ? L"INTERACTIVE" : L"NON-INTERACTIVE"); From ab4dde6c7f4397317ca27d6465d54c89e649eea5 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:28:18 -0700 Subject: [PATCH 58/79] Revert "Handling EPERM in terminal_give_to_job()" This reverts commit bdcd451030998b3c0081edf523e35d20d8a20f2c. It was meant for the major branch. --- src/proc.cpp | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-) diff --git a/src/proc.cpp b/src/proc.cpp index a6ee9e319..f6d338977 100644 --- a/src/proc.cpp +++ b/src/proc.cpp @@ -789,9 +789,6 @@ bool terminal_give_to_job(job_t *j, int cont) { return true; } - //Previously, terminal_give_to_job was being called for each process in a job, hence all the comments - //and warnings below. It is now only called for the first process in a job.t d - //it may not be safe to call tcsetpgrp if we've already done so, as at that point we are no longer //the controlling process group for the terminal and no longer have permission to set the process //group that is in control, causing tcsetpgrp to return EPERM, even though that's not the documented @@ -801,27 +798,18 @@ bool terminal_give_to_job(job_t *j, int cont) { //for SIGTTOU are installed. Read: http://curiousthing.org/sigttin-sigttou-deep-dive-linux //In all cases, our goal here was just to hand over control of the terminal to this process group, //which is a no-op if it's already been done. - auto previous_owner = tcgetpgrp(STDIN_FILENO); - if (previous_owner == j->pgid) { + if (tcgetpgrp(STDIN_FILENO) == j->pgid) { debug(2, L"Process group %d already has control of terminal\n", j->pgid); return true; } + debug(4, L"Attempting bring process group to foreground via tcsetpgrp for job->pgid %d\n", j->pgid); debug(4, L"caller session id: %d, pgid %d has session id: %d\n", getsid(0), j->pgid, getsid(j->pgid)); - //the tcsetpgrp(2) man page says that EPERM is thrown if "pgrp has a supported value, but is not the - //process group ID of a process in the same session as the calling process." - //Since we _guarantee_ that this isn't the case (the child calls setpgid before it calls SIGSTOP, and - //the child was created in the same session as us), it seems that EPERM is being thrown because of an - //caching issue - the call to tcsetpgrp isn't seeing the newly-created process group just yet. On this - //developer's test machine (WSL running Linux 4.4.0), EPERM does indeed disappear on retry. The important - //thing is that we can guarantee the process isn't going to exit while we wait (which would cause us to - //possibly block indefinitely). - signal_block(true); int result = -1; errno = EINTR; - while (result == -1 && (errno == EINTR || errno == EPERM)) { + while (result == -1 && errno == EINTR) { result = tcsetpgrp(STDIN_FILENO, j->pgid); } if (result == -1) { From 9af7720f19f87f48d73edcda7c27069f8e85c598 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:28:28 -0700 Subject: [PATCH 59/79] Revert "Fixed exec failure regression" This reverts commit 8b8a21dcad2505aec15753f308045ae739eda918. It was meant for the major branch. --- src/postfork.cpp | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/postfork.cpp b/src/postfork.cpp index 999324bd1..6832d748e 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -250,9 +250,8 @@ static int handle_child_io(const io_chain_t &io_chain) { } int setup_child_process(job_t *j, process_t *p, const io_chain_t &io_chain) { - bool ok = true; + bool ok = false; - //p is zero when EXEC_INTERNAL if (p) { ok = child_set_group(j, p); } From 8d53b72e46c8501bb0386c8941de5bfc58091172 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:28:38 -0700 Subject: [PATCH 60/79] Revert "Set child process group in case of posix_spawn" This reverts commit 9f2addcf27f5f62b367ab47e3b31ccffa68e5cc6. It was meant for the major branch. --- src/exec.cpp | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 376818dd4..a16d6ac28 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -515,7 +515,6 @@ void exec_job(parser_t &parser, job_t *j) { const bool pipes_to_next_command = !p->is_last_in_job; //set to true if we end up forking for this process bool child_forked = false; - bool child_spawned = false; // The pipes the current process write to and read from. Unfortunately these can't be just // allocated on the stack, since j->io wants shared_ptr. @@ -1093,9 +1092,6 @@ void exec_job(parser_t &parser, job_t *j) { job_mark_process_as_failed(j, p); exec_error = true; } - else { - child_spawned = true; - } } else #endif { @@ -1137,11 +1133,7 @@ void exec_job(parser_t &parser, job_t *j) { } } - if (child_spawned) { - child_set_group(j, p); - set_child_group(j, p->pid); - } - else if (child_forked) { + if (child_forked) { //we have to wait to ensure the child has set their progress group and is in SIGSTOP state //otherwise, we can later call SIGCONT before they call SIGSTOP and they'll be blocked indefinitely. pid_t pid_status{}; From 9d5990eda7a2ae0c58dd690472c06f0607ef0dcf Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:28:48 -0700 Subject: [PATCH 61/79] Revert "Changed how process groups are assigned to child processes" This reverts commit 25afc9b377de5255673276314bf9d26a3b43cbab. It was meant for the major branch. --- src/exec.cpp | 122 +++++++++++++++++++++++++++-------------------- src/postfork.cpp | 116 +++++++++++++++++++++----------------------- src/postfork.h | 3 +- src/proc.cpp | 2 +- src/proc.h | 2 - 5 files changed, 125 insertions(+), 120 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index a16d6ac28..ab13a5513 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -21,7 +21,6 @@ #include #include #include -#include #include #include #include @@ -384,6 +383,21 @@ void exec_job(parser_t &parser, job_t *j) { return; } + auto unblock_pid = [] (pid_t blocked_pid) { + //this is correct, except there's a race condition if the child hasn't yet SIGSTOP'd + //in that case, they never receive the SIGCONT + pid_t pid_status{}; + if (waitpid(blocked_pid, &pid_status, WUNTRACED) != -1) { + // if (WIFSTOPPED(pid_status)) { + debug(2, L"Unblocking process %d.\n", blocked_pid); + kill(blocked_pid, SIGCONT); + return true; + // } + } + debug(2, L"waitpid call in unblock_pid failed!\n"); + return false; + }; + debug(4, L"Exec job '%ls' with id %d", j->command_wcstr(), j->job_id); // Verify that all IO_BUFFERs are output. We used to support a (single, hacked-in) magical input @@ -471,14 +485,14 @@ void exec_job(parser_t &parser, job_t *j) { if (keepalive.pid == 0) { // Child keepalive.pid = getpid(); - child_set_group(j, &keepalive); + set_child_group(j, &keepalive, 1); pause(); exit_without_destructors(0); } else { // Parent debug(2, L"Fork #%d, pid %d: keepalive fork for '%ls'", g_fork_count, keepalive.pid, j->command_wcstr()); - set_child_group(j, keepalive.pid); + set_child_group(j, &keepalive, 0); } } @@ -513,8 +527,7 @@ void exec_job(parser_t &parser, job_t *j) { // See if we need a pipe. const bool pipes_to_next_command = !p->is_last_in_job; - //set to true if we end up forking for this process - bool child_forked = false; + bool command_blocked = false; // The pipes the current process write to and read from. Unfortunately these can't be just // allocated on the stack, since j->io wants shared_ptr. @@ -619,16 +632,12 @@ void exec_job(parser_t &parser, job_t *j) { // This is the IO buffer we use for storing the output of a block or function when it is in // a pipeline. shared_ptr block_output_io_buffer; - - auto unblock_previous = [&j] () { - //we've already called waitpid after forking the child, so we've guaranteed that they're - //already SIGSTOP'd. Otherwise we'd be risking a deadlock because we can call SIGCONT before - //they've actually stopped, and they'll remain suspended indefinitely. + auto unblock_previous = [&] () { if (blocked_pid != -1) { - debug(2, L"Unblocking process %d.\n", blocked_pid); - kill(blocked_pid, SIGCONT); + //now that next command in the chain has been started, unblock the previous command + unblock_pid(blocked_pid); + blocked_pid = -1; } - blocked_pid = -1; }; // This is the io_streams we pass to internal builtins. @@ -884,10 +893,14 @@ void exec_job(parser_t &parser, job_t *j) { // This is the child process. Write out the contents of the pipeline. p->pid = getpid(); setup_child_process(j, p, process_net_io_chain); - // Make child processes pause after executing setup_child_process() to give down-chain - // commands in the job a chance to join their process group and read their pipes. + // Start child processes that are part of a chain in a stopped state + // to ensure that they are still running when the next command in the + // chain is started. // The process will be resumed when the next command in the chain is started. - kill(p->pid, SIGSTOP); + // Note that this may span multiple jobs, as jobs can read from each other. + if (pipes_to_next_command) { + kill(p->pid, SIGSTOP); + } exec_write_and_exit(block_output_io_buffer->fd, buffer, count, status); } else { @@ -895,9 +908,14 @@ void exec_job(parser_t &parser, job_t *j) { // possibly give it control over the terminal. debug(2, L"Fork #%d, pid %d: internal block or function for '%ls'", g_fork_count, pid, p->argv0()); - child_forked = true; - debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); + if (pipes_to_next_command) { + //it actually blocked itself after forking above, but print in here for output + //synchronization & so we can assign command_blocked in the correct address space + debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); + command_blocked = true; + } p->pid = pid; + set_child_group(j, p, 0); } } else { @@ -1012,11 +1030,14 @@ void exec_job(parser_t &parser, job_t *j) { // stdout and stderr, and then exit. p->pid = getpid(); setup_child_process(j, p, process_net_io_chain); - // Make child processes pause after executing setup_child_process() to give down-chain - // commands in the job a chance to join their process group and read their pipes. + // Start child processes that are part of a chain in a stopped state + // to ensure that they are still running when the next command in the + // chain is started. // The process will be resumed when the next command in the chain is started. - kill(p->pid, SIGSTOP); - + // Note that this may span multiple jobs, as jobs can read from each other. + if (pipes_to_next_command) { + kill(p->pid, SIGSTOP); + } do_builtin_io(outbuff, outbuff_len, errbuff, errbuff_len); exit_without_destructors(p->status); } else { @@ -1024,9 +1045,15 @@ void exec_job(parser_t &parser, job_t *j) { // possibly give it control over the terminal. debug(2, L"Fork #%d, pid %d: internal builtin for '%ls'", g_fork_count, pid, p->argv0()); - child_forked = true; - debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); + if (pipes_to_next_command) { + //it actually blocked itself after forking above, but print in here for output + //synchronization & so we can assign command_blocked in the correct address space + debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); + command_blocked = true; + } p->pid = pid; + + set_child_group(j, p, 0); } } @@ -1100,10 +1127,15 @@ void exec_job(parser_t &parser, job_t *j) { // This is the child process. p->pid = getpid(); setup_child_process(j, p, process_net_io_chain); - // Make child processes pause after executing setup_child_process() to give down-chain - // commands in the job a chance to join their process group and read their pipes. + + // Start child processes that are part of a chain in a stopped state + // to ensure that they are still running when the next command in the + // chain is started. // The process will be resumed when the next command in the chain is started. - kill(p->pid, SIGSTOP); + // Note that this may span multiple jobs, as jobs can read from each other. + if (pipes_to_next_command) { + kill(p->pid, SIGSTOP); + } safe_launch_process(p, actual_cmd, argv, envv); // safe_launch_process _never_ returns... @@ -1115,14 +1147,19 @@ void exec_job(parser_t &parser, job_t *j) { job_mark_process_as_failed(j, p); exec_error = true; } - child_forked = true; - debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); + if (pipes_to_next_command) { + //it actually blocked itself after forking above, but print in here for output + //synchronization & so we can assign command_blocked in the correct address space + debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); + command_blocked = true; + } } } // This is the parent process. Store away information on the child, and possibly // fice it control over the terminal. p->pid = pid; + set_child_group(j, p, 0); break; } @@ -1133,31 +1170,10 @@ void exec_job(parser_t &parser, job_t *j) { } } - if (child_forked) { - //we have to wait to ensure the child has set their progress group and is in SIGSTOP state - //otherwise, we can later call SIGCONT before they call SIGSTOP and they'll be blocked indefinitely. - pid_t pid_status{}; - if (waitpid(p->pid, &pid_status, WUNTRACED) != -1) { - //the child is SIGSTOP'd and is guaranteed to have called child_set_group() at this point. - set_child_group(j, p->pid); //update our own process group info to match - //we are not unblocking the child via SIGCONT just yet to give the next process a chance to open - //the pipes and join the process group. We only unblock the last process in the job chain because - //no one awaits it. - if (!pipes_to_next_command) - { - kill(p->pid, SIGCONT); - } - } - else { - debug(2, L"waitpid(%d) call in unblock_pid failed:!\n", p->pid); - wperror(L"waitpid"); - } - } - //if the command we ran _before_ this one was SIGSTOP'd to let this one catch up, unblock it now. - //this must be after the wait_pid on the process we just started, if any. - unblock_previous(); - if (child_forked) { + //if the command we ran before this one was SIGSTOP'd to let this one catch up, unblock it now. + unblock_previous(); + if (command_blocked) { //store the newly-blocked command's PID so that it can be SIGCONT'd once the next process //in the chain is started. That may be in this job or in another job. blocked_pid = p->pid; diff --git a/src/postfork.cpp b/src/postfork.cpp index 6832d748e..1270dfee0 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -59,18 +59,14 @@ static void debug_safe_int(int level, const char *format, int val) { debug_safe(level, format, buff); } -/// Called only by the child to set its own process group (possibly creating a new group in the -/// process if it is the first in a JOB_CONTROL job. The parent will wait for this to finish. -/// A process that isn't already in control of the terminal can't give itself control of the -/// terminal without hanging, but it's not right for the child to try and give itself control -/// from the very beginning because the parent may not have gotten around to doing so yet. Let -/// the parent figure it out; if the child doesn't have terminal control and it later tries to -/// read from the terminal, the kernel will send it SIGTTIN and it'll hang anyway. -/// The key here is that the parent should transfer control of the terminal (if appropriate) -/// prior to sending the child SIGCONT to wake it up to exec. +/// This function should be called by both the parent process and the child right after fork() has +/// been called. If job control is enabled, the child is put in the jobs group, and if the child is +/// also in the foreground, it is also given control of the terminal. When called in the parent +/// process, this function may fail, since the child might have already finished and called exit. +/// The parent process may safely ignore the exit status of this call. /// /// Returns true on sucess, false on failiure. -bool child_set_group(job_t *j, process_t *p) { +bool set_child_group(job_t *j, process_t *p, int print_errors) { bool retval = true; if (j->get_flag(JOB_CONTROL)) { @@ -78,53 +74,32 @@ bool child_set_group(job_t *j, process_t *p) { if (j->pgid == -2) { j->pgid = p->pid; } - int failure = setpgid(p->pid, j->pgid); - // TODO: Figure out why we're testing whether the pgid is correct after attempting to - // set it failed. This was added in commit 4e912ef8 from 2012-02-27. - failure = failure && getpgid(p->pid) != j->pgid; - if (failure) { //!OCLINT(collapsible if statements) - char pid_buff[128]; - char job_id_buff[128]; - char getpgid_buff[128]; - char job_pgid_buff[128]; - char argv0[64]; - char command[64]; - format_long_safe(pid_buff, p->pid); - format_long_safe(job_id_buff, j->job_id); - format_long_safe(getpgid_buff, getpgid(p->pid)); - format_long_safe(job_pgid_buff, j->pgid); - narrow_string_safe(argv0, p->argv0()); - narrow_string_safe(command, j->command_wcstr()); + if (setpgid(p->pid, j->pgid)) { //!OCLINT(collapsible if statements) + // TODO: Figure out why we're testing whether the pgid is correct after attempting to + // set it failed. This was added in commit 4e912ef8 from 2012-02-27. + if (getpgid(p->pid) != j->pgid && print_errors) { + char pid_buff[128]; + char job_id_buff[128]; + char getpgid_buff[128]; + char job_pgid_buff[128]; + char argv0[64]; + char command[64]; - debug_safe( - 1, "Could not send own process %s, '%s' in job %s, '%s' from group %s to group %s", - pid_buff, argv0, job_id_buff, command, getpgid_buff, job_pgid_buff); + format_long_safe(pid_buff, p->pid); + format_long_safe(job_id_buff, j->job_id); + format_long_safe(getpgid_buff, getpgid(p->pid)); + format_long_safe(job_pgid_buff, j->pgid); + narrow_string_safe(argv0, p->argv0()); + narrow_string_safe(command, j->command_wcstr()); - safe_perror("setpgid"); - retval = false; - } - } else { - //this is probably stays unused in the child - j->pgid = getpgrp(); - } + debug_safe( + 1, "Could not send process %s, '%s' in job %s, '%s' from group %s to group %s", + pid_buff, argv0, job_id_buff, command, getpgid_buff, job_pgid_buff); - return retval; -} - -/// Called only by the parent only after a child forks and successfully calls child_set_group, guaranteeing -/// the job control process group has been created and that the child belongs to the correct process group. -/// Here we can update our job_t structure to reflect the correct process group in the case of JOB_CONTROL, -/// and we can give the new process group control of the terminal if it's to run in the foreground. Note that -/// we can guarantee the child won't try to read from the terminal before we've had a chance to run this code, -/// because we haven't woken them up with a SIGCONT yet. -bool set_child_group(job_t *j, pid_t child_pid) { - bool retval = true; - - if (j->get_flag(JOB_CONTROL)) { - // New jobs have the pgid set to -2 - if (j->pgid == -2) { - j->pgid = child_pid; + safe_perror("setpgid"); + retval = false; + } } } else { j->pgid = getpgrp(); @@ -132,16 +107,33 @@ bool set_child_group(job_t *j, pid_t child_pid) { if (j->get_flag(JOB_TERMINAL) && j->get_flag(JOB_FOREGROUND)) { //!OCLINT(early exit) if (tcgetpgrp(STDIN_FILENO) == j->pgid) { - //we've already assigned the process group control of the terminal when the first process in the job - //was started. There's no need to do so again, and on some platforms this can cause an EPERM error. - //In addition, if we've given control of the terminal to a process group, attempting to call tcsetpgrp - //from the background will cause SIGTTOU to be sent to everything in our process group (unless we - //handle it).. debug(4, L"Process group %d already has control of terminal\n", j->pgid); } else { - //no need to duplicate the code here, a function already exists that does just this - retval = terminal_give_to_job(j, false /*new job, so not continuing*/); + debug(4, L"Attempting bring process group to foreground via tcsetpgrp for job->pgid %d\n", j->pgid); + debug(4, L"caller session id: %d, pgid %d has session id: %d\n", getsid(0), j->pgid, getsid(j->pgid)); + int result = -1; + errno = EINTR; + while (result == -1 && errno == EINTR) { + signal_block(true); + result = tcsetpgrp(STDIN_FILENO, j->pgid); + signal_unblock(true); + } + if (result == -1) { + if (errno == ENOTTY) redirect_tty_output(); + if (print_errors) { + char job_id_buff[64]; + char command_buff[64]; + char job_pgid_buff[128]; + format_long_safe(job_id_buff, j->job_id); + narrow_string_safe(command_buff, j->command_wcstr()); + format_long_safe(job_pgid_buff, j->pgid); + debug_safe(1, "Could not send job %s ('%s') with pgid %s to foreground", job_id_buff, + command_buff, job_pgid_buff); + safe_perror("tcsetpgrp"); + retval = false; + } + } } } @@ -250,10 +242,10 @@ static int handle_child_io(const io_chain_t &io_chain) { } int setup_child_process(job_t *j, process_t *p, const io_chain_t &io_chain) { - bool ok = false; + bool ok = true; if (p) { - ok = child_set_group(j, p); + ok = set_child_group(j, p, 1); } if (ok) { diff --git a/src/postfork.h b/src/postfork.h index a90fd377b..0a82a4553 100644 --- a/src/postfork.h +++ b/src/postfork.h @@ -18,8 +18,7 @@ class io_chain_t; class job_t; class process_t; -bool set_child_group(job_t *j, pid_t child_pid); //called by parent -bool child_set_group(job_t *j, process_t *p); //called by child +bool set_child_group(job_t *j, process_t *p, int print_errors); /// Initialize a new child process. This should be called right away after forking in the child /// process. If job control is enabled for this job, the process is put in the process group of the diff --git a/src/proc.cpp b/src/proc.cpp index f6d338977..061aa2c96 100644 --- a/src/proc.cpp +++ b/src/proc.cpp @@ -782,7 +782,7 @@ static void read_try(job_t *j) { /// \param j The job to give the terminal to. /// \param cont If this variable is set, we are giving back control to a job that has previously /// been stopped. In that case, we need to set the terminal attributes to those saved in the job. -bool terminal_give_to_job(job_t *j, int cont) { +static bool terminal_give_to_job(job_t *j, int cont) { errno = 0; if (j->pgid == 0) { debug(2, "terminal_give_to_job() returning early due to no process group"); diff --git a/src/proc.h b/src/proc.h index b57b89c25..874fb8230 100644 --- a/src/proc.h +++ b/src/proc.h @@ -365,5 +365,3 @@ void proc_pop_interactive(); int proc_format_status(int status); #endif - -bool terminal_give_to_job(job_t *j, int cont); From 559f8d551197d95fcdf853c488d78eefcc1a0549 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:28:59 -0700 Subject: [PATCH 62/79] Revert "Don't indiscriminately unblock previous cmd for internal builtin/functions" This reverts commit c81cf56c0b6355e9d5e50d74e56fc00e1039d40a. It was meant for the major branch. --- src/exec.cpp | 35 +++++++++++++++-------------------- 1 file changed, 15 insertions(+), 20 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index ab13a5513..1c811af85 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -592,6 +592,16 @@ void exec_job(parser_t &parser, job_t *j) { } env_export_arr(); } + else { + // Since the main loop itself is going to be (potentially) reading from the previous + // process, we need to unblock it here instead of after "executing" the next process + // in the chain. + if (blocked_pid != -1) { + //now that next command in the chain has been started, unblock the previous command + unblock_pid(blocked_pid); + blocked_pid = -1; + } + } // Set up fds that will be used in the pipe. if (pipes_to_next_command) { @@ -632,23 +642,12 @@ void exec_job(parser_t &parser, job_t *j) { // This is the IO buffer we use for storing the output of a block or function when it is in // a pipeline. shared_ptr block_output_io_buffer; - auto unblock_previous = [&] () { - if (blocked_pid != -1) { - //now that next command in the chain has been started, unblock the previous command - unblock_pid(blocked_pid); - blocked_pid = -1; - } - }; // This is the io_streams we pass to internal builtins. std::unique_ptr builtin_io_streams(new io_streams_t(stdout_read_limit)); switch (p->type) { case INTERNAL_FUNCTION: { - //internal function never forks - //unblock previous (if any) to prevent hang - unblock_previous(); - // Calls to function_get_definition might need to source a file as a part of // autoloading, hence there must be no blocks. signal_unblock(); @@ -826,9 +825,6 @@ void exec_job(parser_t &parser, job_t *j) { signal_unblock(); - //main loop is performing a blocking read from previous command's output - //make sure read source is not blocked - unblock_previous(); p->status = builtin_run(parser, p->get_argv(), *builtin_io_streams); signal_block(); @@ -949,10 +945,6 @@ void exec_job(parser_t &parser, job_t *j) { // output, so that we can truncate the file. Does not apply to /dev/null. bool must_fork = redirection_is_to_real_file(stdout_io.get()) || redirection_is_to_real_file(stderr_io.get()); - if (!must_fork) { - //we are handling reads directly in the main loop. Make sure source is unblocked. - unblock_previous(); - } if (!must_fork && p->is_last_in_job) { const bool stdout_is_to_buffer = stdout_io && stdout_io->io_mode == IO_BUFFER; const bool no_stdout_output = stdout_buffer.empty(); @@ -1170,9 +1162,12 @@ void exec_job(parser_t &parser, job_t *j) { } } + if (blocked_pid != -1) { + //now that next command in the chain has been started, unblock the previous command + unblock_pid(blocked_pid); + blocked_pid = -1; + } - //if the command we ran before this one was SIGSTOP'd to let this one catch up, unblock it now. - unblock_previous(); if (command_blocked) { //store the newly-blocked command's PID so that it can be SIGCONT'd once the next process //in the chain is started. That may be in this job or in another job. From 158b946eaccdd624d71e1bba1b03b303e2d92f23 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:29:09 -0700 Subject: [PATCH 63/79] Revert "Don't attempt to unconditionally tcsetpgrp" This reverts commit 0e9177b59090e1f7bab048436b7c625ef3544947. It was meant for the major branch. --- src/postfork.cpp | 43 +++++++++++++++++-------------------------- src/proc.cpp | 19 +------------------ 2 files changed, 18 insertions(+), 44 deletions(-) diff --git a/src/postfork.cpp b/src/postfork.cpp index 1270dfee0..9f29963b0 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -106,33 +106,24 @@ bool set_child_group(job_t *j, process_t *p, int print_errors) { } if (j->get_flag(JOB_TERMINAL) && j->get_flag(JOB_FOREGROUND)) { //!OCLINT(early exit) - if (tcgetpgrp(STDIN_FILENO) == j->pgid) { - debug(4, L"Process group %d already has control of terminal\n", j->pgid); + int result = -1; + errno = EINTR; + while (result == -1 && errno == EINTR) { + signal_block(true); + result = tcsetpgrp(STDIN_FILENO, j->pgid); + signal_unblock(true); } - else { - debug(4, L"Attempting bring process group to foreground via tcsetpgrp for job->pgid %d\n", j->pgid); - debug(4, L"caller session id: %d, pgid %d has session id: %d\n", getsid(0), j->pgid, getsid(j->pgid)); - int result = -1; - errno = EINTR; - while (result == -1 && errno == EINTR) { - signal_block(true); - result = tcsetpgrp(STDIN_FILENO, j->pgid); - signal_unblock(true); - } - if (result == -1) { - if (errno == ENOTTY) redirect_tty_output(); - if (print_errors) { - char job_id_buff[64]; - char command_buff[64]; - char job_pgid_buff[128]; - format_long_safe(job_id_buff, j->job_id); - narrow_string_safe(command_buff, j->command_wcstr()); - format_long_safe(job_pgid_buff, j->pgid); - debug_safe(1, "Could not send job %s ('%s') with pgid %s to foreground", job_id_buff, - command_buff, job_pgid_buff); - safe_perror("tcsetpgrp"); - retval = false; - } + if (result == -1) { + if (errno == ENOTTY) redirect_tty_output(); + if (print_errors) { + char job_id_buff[64]; + char command_buff[64]; + format_long_safe(job_id_buff, j->job_id); + narrow_string_safe(command_buff, j->command_wcstr()); + debug_safe(1, "Could not send job %s ('%s') to foreground", job_id_buff, + command_buff); + safe_perror("tcsetpgrp"); + retval = false; } } } diff --git a/src/proc.cpp b/src/proc.cpp index 061aa2c96..063f01feb 100644 --- a/src/proc.cpp +++ b/src/proc.cpp @@ -789,23 +789,6 @@ static bool terminal_give_to_job(job_t *j, int cont) { return true; } - //it may not be safe to call tcsetpgrp if we've already done so, as at that point we are no longer - //the controlling process group for the terminal and no longer have permission to set the process - //group that is in control, causing tcsetpgrp to return EPERM, even though that's not the documented - //behavior in tcsetpgrp(3), which instead says other bad things will happen (it says SIGTTOU will be - //sent to all members of the background *calling* process group, but it's more complicated than that, - //SIGTTOU may or may not be sent depending on the TTY configuration and whether or not signal handlers - //for SIGTTOU are installed. Read: http://curiousthing.org/sigttin-sigttou-deep-dive-linux - //In all cases, our goal here was just to hand over control of the terminal to this process group, - //which is a no-op if it's already been done. - if (tcgetpgrp(STDIN_FILENO) == j->pgid) { - debug(2, L"Process group %d already has control of terminal\n", j->pgid); - return true; - } - - debug(4, L"Attempting bring process group to foreground via tcsetpgrp for job->pgid %d\n", j->pgid); - debug(4, L"caller session id: %d, pgid %d has session id: %d\n", getsid(0), j->pgid, getsid(j->pgid)); - signal_block(true); int result = -1; errno = EINTR; @@ -814,7 +797,7 @@ static bool terminal_give_to_job(job_t *j, int cont) { } if (result == -1) { if (errno == ENOTTY) redirect_tty_output(); - debug(1, _(L"terminal_give_to_job(): Could not send job %d ('%ls') with pgid %d to foreground"), j->job_id, j->command_wcstr(), j->pgid); + debug(1, _(L"Could not send job %d ('%ls') to foreground"), j->job_id, j->command_wcstr()); wperror(L"tcsetpgrp"); signal_unblock(true); return false; From 8ebba2066d8149d0f1fe9d0282773d757e535453 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:29:22 -0700 Subject: [PATCH 64/79] Revert "Fixed race condition in new job control synchronization" This reverts commit 87394a9e0b1aba875b2ccb1c03ba0314ba41ff9e. It was meant for the major branch. --- src/exec.cpp | 70 ++++++---------------------------------------------- 1 file changed, 8 insertions(+), 62 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 1c811af85..34657b5ce 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -15,7 +15,6 @@ #endif #include #include -#include #include #include @@ -383,21 +382,6 @@ void exec_job(parser_t &parser, job_t *j) { return; } - auto unblock_pid = [] (pid_t blocked_pid) { - //this is correct, except there's a race condition if the child hasn't yet SIGSTOP'd - //in that case, they never receive the SIGCONT - pid_t pid_status{}; - if (waitpid(blocked_pid, &pid_status, WUNTRACED) != -1) { - // if (WIFSTOPPED(pid_status)) { - debug(2, L"Unblocking process %d.\n", blocked_pid); - kill(blocked_pid, SIGCONT); - return true; - // } - } - debug(2, L"waitpid call in unblock_pid failed!\n"); - return false; - }; - debug(4, L"Exec job '%ls' with id %d", j->command_wcstr(), j->job_id); // Verify that all IO_BUFFERs are output. We used to support a (single, hacked-in) magical input @@ -592,16 +576,6 @@ void exec_job(parser_t &parser, job_t *j) { } env_export_arr(); } - else { - // Since the main loop itself is going to be (potentially) reading from the previous - // process, we need to unblock it here instead of after "executing" the next process - // in the chain. - if (blocked_pid != -1) { - //now that next command in the chain has been started, unblock the previous command - unblock_pid(blocked_pid); - blocked_pid = -1; - } - } // Set up fds that will be used in the pipe. if (pipes_to_next_command) { @@ -889,14 +863,6 @@ void exec_job(parser_t &parser, job_t *j) { // This is the child process. Write out the contents of the pipeline. p->pid = getpid(); setup_child_process(j, p, process_net_io_chain); - // Start child processes that are part of a chain in a stopped state - // to ensure that they are still running when the next command in the - // chain is started. - // The process will be resumed when the next command in the chain is started. - // Note that this may span multiple jobs, as jobs can read from each other. - if (pipes_to_next_command) { - kill(p->pid, SIGSTOP); - } exec_write_and_exit(block_output_io_buffer->fd, buffer, count, status); } else { @@ -904,12 +870,6 @@ void exec_job(parser_t &parser, job_t *j) { // possibly give it control over the terminal. debug(2, L"Fork #%d, pid %d: internal block or function for '%ls'", g_fork_count, pid, p->argv0()); - if (pipes_to_next_command) { - //it actually blocked itself after forking above, but print in here for output - //synchronization & so we can assign command_blocked in the correct address space - debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); - command_blocked = true; - } p->pid = pid; set_child_group(j, p, 0); } @@ -1022,14 +982,6 @@ void exec_job(parser_t &parser, job_t *j) { // stdout and stderr, and then exit. p->pid = getpid(); setup_child_process(j, p, process_net_io_chain); - // Start child processes that are part of a chain in a stopped state - // to ensure that they are still running when the next command in the - // chain is started. - // The process will be resumed when the next command in the chain is started. - // Note that this may span multiple jobs, as jobs can read from each other. - if (pipes_to_next_command) { - kill(p->pid, SIGSTOP); - } do_builtin_io(outbuff, outbuff_len, errbuff, errbuff_len); exit_without_destructors(p->status); } else { @@ -1037,12 +989,6 @@ void exec_job(parser_t &parser, job_t *j) { // possibly give it control over the terminal. debug(2, L"Fork #%d, pid %d: internal builtin for '%ls'", g_fork_count, pid, p->argv0()); - if (pipes_to_next_command) { - //it actually blocked itself after forking above, but print in here for output - //synchronization & so we can assign command_blocked in the correct address space - debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); - command_blocked = true; - } p->pid = pid; set_child_group(j, p, 0); @@ -1133,18 +1079,18 @@ void exec_job(parser_t &parser, job_t *j) { // safe_launch_process _never_ returns... DIE("safe_launch_process should not have returned"); } else { - debug(2, L"Fork #%d, pid %d: external command '%s' from '%ls'", - g_fork_count, pid, p->argv0(), file ? file : L""); - if (pid < 0) { - job_mark_process_as_failed(j, p); - exec_error = true; - } if (pipes_to_next_command) { //it actually blocked itself after forking above, but print in here for output //synchronization & so we can assign command_blocked in the correct address space debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); command_blocked = true; } + debug(2, L"Fork #%d, pid %d: external command '%s' from '%ls'", + g_fork_count, pid, p->argv0(), file ? file : L""); + if (pid < 0) { + job_mark_process_as_failed(j, p); + exec_error = true; + } } } @@ -1164,10 +1110,10 @@ void exec_job(parser_t &parser, job_t *j) { if (blocked_pid != -1) { //now that next command in the chain has been started, unblock the previous command - unblock_pid(blocked_pid); + debug(2, L"Unblocking process %d.\n", blocked_pid); + kill(blocked_pid, SIGCONT); blocked_pid = -1; } - if (command_blocked) { //store the newly-blocked command's PID so that it can be SIGCONT'd once the next process //in the chain is started. That may be in this job or in another job. From e0ccb8c48e2c11aca56ac0d60d73299e975c305e Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:29:33 -0700 Subject: [PATCH 65/79] Revert "Improved blocked prcoess comments, clarified job vs command chain" This reverts commit dfac81803b0da9fc1fd1d71f2f01abb1909dfc07. It was meant for the major branch. --- src/exec.cpp | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 34657b5ce..afa69b2a0 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -1064,13 +1064,13 @@ void exec_job(parser_t &parser, job_t *j) { if (pid == 0) { // This is the child process. p->pid = getpid(); + // the process will be resumed by the shell when the next command in the + // chain is started setup_child_process(j, p, process_net_io_chain); - // Start child processes that are part of a chain in a stopped state + // start child processes that are part of a job in a stopped state // to ensure that they are still running when the next command in the // chain is started. - // The process will be resumed when the next command in the chain is started. - // Note that this may span multiple jobs, as jobs can read from each other. if (pipes_to_next_command) { kill(p->pid, SIGSTOP); } @@ -1081,7 +1081,7 @@ void exec_job(parser_t &parser, job_t *j) { } else { if (pipes_to_next_command) { //it actually blocked itself after forking above, but print in here for output - //synchronization & so we can assign command_blocked in the correct address space + //synchronization and so we can assign blocked_pid in the correct address space debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); command_blocked = true; } @@ -1115,8 +1115,6 @@ void exec_job(parser_t &parser, job_t *j) { blocked_pid = -1; } if (command_blocked) { - //store the newly-blocked command's PID so that it can be SIGCONT'd once the next process - //in the chain is started. That may be in this job or in another job. blocked_pid = p->pid; } From 950c7ef2ebf518eeb81ba87b980609627e39b275 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:29:41 -0700 Subject: [PATCH 66/79] Revert "fixup! Using SIGSTOP/SIGCONT instead of mmap & sem_t to synchronize jobs" This reverts commit f653fbfaf4bfea2a21e7d9b6dfebc9ceaa9ca58d. It was meant for the major branch. --- src/exec.cpp | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/exec.cpp b/src/exec.cpp index afa69b2a0..c5c60ab9c 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -513,6 +513,10 @@ void exec_job(parser_t &parser, job_t *j) { const bool pipes_to_next_command = !p->is_last_in_job; bool command_blocked = false; + //these semaphores will be used to make sure the first process lives long enough for the + //next process in the chain to open its handles, process group, etc. + //this child will block on this one, the next child will have to call sem_post against it. + // The pipes the current process write to and read from. Unfortunately these can't be just // allocated on the stack, since j->io wants shared_ptr. // From 635365654d192ec7f5034eb87b5e27af39f83117 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:29:50 -0700 Subject: [PATCH 67/79] Revert "Fixed cases where first command in chain would stay blocked" This reverts commit fb13b370e26b66fee6f662ac58e06ed04b47fbc9. It was meant for the major branch. --- src/exec.cpp | 37 ++++++++++++++----------------------- 1 file changed, 14 insertions(+), 23 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index c5c60ab9c..35b007fed 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -493,7 +493,7 @@ void exec_job(parser_t &parser, job_t *j) { // // We are careful to set these to -1 when closed, so if we exit the loop abruptly, we can still // close them. - static pid_t blocked_pid = -1; + int last_pid = -1; int pipe_current_read = -1, pipe_current_write = -1, pipe_next_read = -1; for (std::unique_ptr &unique_p : j->processes) { if (exec_error) { @@ -511,7 +511,6 @@ void exec_job(parser_t &parser, job_t *j) { // See if we need a pipe. const bool pipes_to_next_command = !p->is_last_in_job; - bool command_blocked = false; //these semaphores will be used to make sure the first process lives long enough for the //next process in the chain to open its handles, process group, etc. @@ -1068,27 +1067,20 @@ void exec_job(parser_t &parser, job_t *j) { if (pid == 0) { // This is the child process. p->pid = getpid(); - // the process will be resumed by the shell when the next command in the - // chain is started - setup_child_process(j, p, process_net_io_chain); - // start child processes that are part of a job in a stopped state // to ensure that they are still running when the next command in the // chain is started. if (pipes_to_next_command) { + debug(3, L"Blocking process %d waiting for next command in chain.\n", p->pid); kill(p->pid, SIGSTOP); } - + // the process will be resumed by the shell when the next command in the + // chain is started + setup_child_process(j, p, process_net_io_chain); safe_launch_process(p, actual_cmd, argv, envv); // safe_launch_process _never_ returns... DIE("safe_launch_process should not have returned"); } else { - if (pipes_to_next_command) { - //it actually blocked itself after forking above, but print in here for output - //synchronization and so we can assign blocked_pid in the correct address space - debug(2, L"Blocking process %d waiting for next command in chain.\n", pid); - command_blocked = true; - } debug(2, L"Fork #%d, pid %d: external command '%s' from '%ls'", g_fork_count, pid, p->argv0(), file ? file : L""); if (pid < 0) { @@ -1112,16 +1104,6 @@ void exec_job(parser_t &parser, job_t *j) { } } - if (blocked_pid != -1) { - //now that next command in the chain has been started, unblock the previous command - debug(2, L"Unblocking process %d.\n", blocked_pid); - kill(blocked_pid, SIGCONT); - blocked_pid = -1; - } - if (command_blocked) { - blocked_pid = p->pid; - } - // Close the pipe the current process uses to read from the previous process_t. if (pipe_current_read >= 0) { exec_close(pipe_current_read); @@ -1133,6 +1115,15 @@ void exec_job(parser_t &parser, job_t *j) { exec_close(pipe_current_write); pipe_current_write = -1; } + + //now that next command in the chain has been started, unblock the previous command + if (last_pid != -1) { + debug(3, L"Unblocking process %d.\n", last_pid); + kill(last_pid, SIGCONT); + } + if (pid != 0) { + last_pid = pid; + } } // Clean up any file descriptors we left open. From 44295186677a753ed22379380ca8278663993e36 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:29:59 -0700 Subject: [PATCH 68/79] Revert "Using SIGSTOP/SIGCONT instead of mmap & sem_t to synchronize jobs" This reverts commit cafd856831507f0599f5ea24231a3ee25f62d174. It was meant for the major branch. --- src/exec.cpp | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 35b007fed..03fc5687a 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -23,6 +23,8 @@ #include #include #include +#include +#include #include "builtin.h" #include "common.h" @@ -493,7 +495,7 @@ void exec_job(parser_t &parser, job_t *j) { // // We are careful to set these to -1 when closed, so if we exit the loop abruptly, we can still // close them. - int last_pid = -1; + sem_t *chained_wait_prev = nullptr; int pipe_current_read = -1, pipe_current_write = -1, pipe_next_read = -1; for (std::unique_ptr &unique_p : j->processes) { if (exec_error) { @@ -515,6 +517,10 @@ void exec_job(parser_t &parser, job_t *j) { //these semaphores will be used to make sure the first process lives long enough for the //next process in the chain to open its handles, process group, etc. //this child will block on this one, the next child will have to call sem_post against it. + sem_t *chained_wait_next = !pipes_to_next_command ? nullptr : (sem_t*)mmap(NULL, sizeof(sem_t*), PROT_READ|PROT_WRITE,MAP_SHARED|MAP_ANONYMOUS,-1, 0); + if (chained_wait_next != nullptr) { + sem_init(chained_wait_next, 1, 0); + } // The pipes the current process write to and read from. Unfortunately these can't be just // allocated on the stack, since j->io wants shared_ptr. @@ -1065,17 +1071,17 @@ void exec_job(parser_t &parser, job_t *j) { { pid = execute_fork(false); if (pid == 0) { + // usleep is a hack that fixes any tcsetpgrp errors caused by race conditions + // usleep(20 * 1000); + // it should no longer be needed with the chained_wait_next code below. + if (chained_wait_next != nullptr) { + debug(3, L"Waiting for next command in chain to start.\n"); + sem_wait(chained_wait_next); + sem_destroy(chained_wait_next); + munmap(chained_wait_next, sizeof(sem_t)); + } // This is the child process. p->pid = getpid(); - // start child processes that are part of a job in a stopped state - // to ensure that they are still running when the next command in the - // chain is started. - if (pipes_to_next_command) { - debug(3, L"Blocking process %d waiting for next command in chain.\n", p->pid); - kill(p->pid, SIGSTOP); - } - // the process will be resumed by the shell when the next command in the - // chain is started setup_child_process(j, p, process_net_io_chain); safe_launch_process(p, actual_cmd, argv, envv); // safe_launch_process _never_ returns... @@ -1117,12 +1123,14 @@ void exec_job(parser_t &parser, job_t *j) { } //now that next command in the chain has been started, unblock the previous command - if (last_pid != -1) { - debug(3, L"Unblocking process %d.\n", last_pid); - kill(last_pid, SIGCONT); + if (chained_wait_prev != nullptr) { + debug(3, L"Unblocking previous command in chain.\n"); + sem_post(chained_wait_prev); + munmap(chained_wait_prev, sizeof(sem_t)); + chained_wait_prev = nullptr; } - if (pid != 0) { - last_pid = pid; + if (chained_wait_next != nullptr) { + chained_wait_prev = chained_wait_next; } } From c304ca1a1ccf2d1032961479a4736bcde8d6ef46 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:30:09 -0700 Subject: [PATCH 69/79] Revert "Explicitly nulling chained_wait_prev after munmap()" This reverts commit 47d8a7e8820382d3957f4bebbac79e5fd24d3d7f. It was meant for the major branch. --- src/exec.cpp | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 03fc5687a..b340b9590 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -1071,9 +1071,8 @@ void exec_job(parser_t &parser, job_t *j) { { pid = execute_fork(false); if (pid == 0) { - // usleep is a hack that fixes any tcsetpgrp errors caused by race conditions + // a hack that fixes any tcsetpgrp errors caused by race conditions // usleep(20 * 1000); - // it should no longer be needed with the chained_wait_next code below. if (chained_wait_next != nullptr) { debug(3, L"Waiting for next command in chain to start.\n"); sem_wait(chained_wait_next); @@ -1127,7 +1126,6 @@ void exec_job(parser_t &parser, job_t *j) { debug(3, L"Unblocking previous command in chain.\n"); sem_post(chained_wait_prev); munmap(chained_wait_prev, sizeof(sem_t)); - chained_wait_prev = nullptr; } if (chained_wait_next != nullptr) { chained_wait_prev = chained_wait_next; From 7776a03fb3084742b3a088b276dc6fd2bf30c066 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:30:17 -0700 Subject: [PATCH 70/79] Revert "Fixes a race condition in output redirection in job chain" This reverts commit cdb72b70248a67920002396179426e45e271d170. It was meant for the major branch. --- src/exec.cpp | 29 ----------------------------- 1 file changed, 29 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index b340b9590..047df6f71 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -23,8 +23,6 @@ #include #include #include -#include -#include #include "builtin.h" #include "common.h" @@ -495,7 +493,6 @@ void exec_job(parser_t &parser, job_t *j) { // // We are careful to set these to -1 when closed, so if we exit the loop abruptly, we can still // close them. - sem_t *chained_wait_prev = nullptr; int pipe_current_read = -1, pipe_current_write = -1, pipe_next_read = -1; for (std::unique_ptr &unique_p : j->processes) { if (exec_error) { @@ -514,14 +511,6 @@ void exec_job(parser_t &parser, job_t *j) { // See if we need a pipe. const bool pipes_to_next_command = !p->is_last_in_job; - //these semaphores will be used to make sure the first process lives long enough for the - //next process in the chain to open its handles, process group, etc. - //this child will block on this one, the next child will have to call sem_post against it. - sem_t *chained_wait_next = !pipes_to_next_command ? nullptr : (sem_t*)mmap(NULL, sizeof(sem_t*), PROT_READ|PROT_WRITE,MAP_SHARED|MAP_ANONYMOUS,-1, 0); - if (chained_wait_next != nullptr) { - sem_init(chained_wait_next, 1, 0); - } - // The pipes the current process write to and read from. Unfortunately these can't be just // allocated on the stack, since j->io wants shared_ptr. // @@ -1071,14 +1060,6 @@ void exec_job(parser_t &parser, job_t *j) { { pid = execute_fork(false); if (pid == 0) { - // a hack that fixes any tcsetpgrp errors caused by race conditions - // usleep(20 * 1000); - if (chained_wait_next != nullptr) { - debug(3, L"Waiting for next command in chain to start.\n"); - sem_wait(chained_wait_next); - sem_destroy(chained_wait_next); - munmap(chained_wait_next, sizeof(sem_t)); - } // This is the child process. p->pid = getpid(); setup_child_process(j, p, process_net_io_chain); @@ -1120,16 +1101,6 @@ void exec_job(parser_t &parser, job_t *j) { exec_close(pipe_current_write); pipe_current_write = -1; } - - //now that next command in the chain has been started, unblock the previous command - if (chained_wait_prev != nullptr) { - debug(3, L"Unblocking previous command in chain.\n"); - sem_post(chained_wait_prev); - munmap(chained_wait_prev, sizeof(sem_t)); - } - if (chained_wait_next != nullptr) { - chained_wait_prev = chained_wait_next; - } } // Clean up any file descriptors we left open. From 7dd2cd7de76ffe1305823d737022a9bf01e1511b Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:30:34 -0700 Subject: [PATCH 71/79] Revert "Revert "finish cleanup of signal blocking code"" This reverts commit 35ee28ff24c7bca933d477029c7f3386cfbaf9a3. It was meant for the major branch. --- src/exec.cpp | 28 +--------------------------- src/fish.cpp | 7 ------- src/history.cpp | 9 --------- src/postfork.cpp | 6 ++---- src/proc.cpp | 16 ++++++++-------- src/reader.cpp | 16 ++-------------- src/signal.cpp | 16 +++------------- src/signal.h | 6 ++---- 8 files changed, 18 insertions(+), 86 deletions(-) diff --git a/src/exec.cpp b/src/exec.cpp index 047df6f71..5af251572 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -36,7 +36,6 @@ #include "postfork.h" #include "proc.h" #include "reader.h" -#include "signal.h" #include "wutil.h" // IWYU pragma: keep /// File descriptor redirection error message. @@ -322,16 +321,12 @@ static void internal_exec_helper(parser_t &parser, const wcstring &def, node_off return; } - signal_unblock(); - if (node_offset == NODE_OFFSET_INVALID) { parser.eval(def, morphed_chain, block_type); } else { parser.eval_block_node(node_offset, morphed_chain, block_type); } - signal_block(); - morphed_chain.clear(); io_cleanup_fds(opened_fds); job_reap(0); @@ -401,10 +396,8 @@ void exec_job(parser_t &parser, job_t *j) { if (j->processes.front()->type == INTERNAL_EXEC) { // Do a regular launch - but without forking first... - signal_block(); - // setup_child_process makes sure signals are properly set up. It will also call - // signal_unblock. + // setup_child_process makes sure signals are properly set up. // PCA This is for handling exec. Passing all_ios here matches what fish 2.0.0 and 1.x did. // It's known to be wrong - for example, it means that redirections bound for subsequent @@ -448,8 +441,6 @@ void exec_job(parser_t &parser, job_t *j) { } } - signal_block(); - // See if we need to create a group keepalive process. This is a process that we create to make // sure that the process group doesn't die accidentally, and is often needed when a // builtin/block/function is inside a pipeline, since that usually means we have to wait for one @@ -620,9 +611,6 @@ void exec_job(parser_t &parser, job_t *j) { switch (p->type) { case INTERNAL_FUNCTION: { - // Calls to function_get_definition might need to source a file as a part of - // autoloading, hence there must be no blocks. - signal_unblock(); const wcstring func_name = p->argv0(); wcstring def; bool function_exists = function_get_definition(func_name, &def); @@ -630,8 +618,6 @@ void exec_job(parser_t &parser, job_t *j) { const std::map inherit_vars = function_get_inherit_vars(func_name); - signal_block(); - if (!function_exists) { debug(0, _(L"Unknown function '%ls'"), p->argv0()); break; @@ -639,13 +625,7 @@ void exec_job(parser_t &parser, job_t *j) { function_block_t *fb = parser.push_block(p, func_name, shadow_scope); - - // Setting variables might trigger an event handler, hence we need to unblock - // signals. - signal_unblock(); function_prepare_environment(func_name, p->get_argv() + 1, inherit_vars); - signal_block(); - parser.forbid_function(func_name); if (!p->is_last_in_job) { @@ -795,12 +775,8 @@ void exec_job(parser_t &parser, job_t *j) { const int fg = j->get_flag(JOB_FOREGROUND); j->set_flag(JOB_FOREGROUND, false); - signal_unblock(); - p->status = builtin_run(parser, p->get_argv(), *builtin_io_streams); - signal_block(); - // Restore the fg flag, which is temporarily set to false during builtin // execution so as not to confuse some job-handling builtins. j->set_flag(JOB_FOREGROUND, fg); @@ -1113,9 +1089,7 @@ void exec_job(parser_t &parser, job_t *j) { kill(keepalive.pid, SIGKILL); } - signal_unblock(); debug(3, L"Job is constructed"); - j->set_flag(JOB_CONSTRUCTED, true); if (!j->get_flag(JOB_FOREGROUND)) { proc_last_bg_pid = j->pgid; diff --git a/src/fish.cpp b/src/fish.cpp index 273c85ab8..096805a6e 100644 --- a/src/fish.cpp +++ b/src/fish.cpp @@ -387,13 +387,6 @@ int main(int argc, char **argv) { const io_chain_t empty_ios; if (read_init(paths)) { - // TODO: Remove this once we're confident that not blocking/unblocking every signal around - // some critical sections is no longer necessary. - env_var_t fish_no_signal_block = env_get_string(L"FISH_NO_SIGNAL_BLOCK"); - if (!fish_no_signal_block.missing_or_empty() && !from_string(fish_no_signal_block)) { - ignore_signal_block = false; - } - // Stomp the exit status of any initialization commands (issue #635). proc_set_last_status(STATUS_CMD_OK); diff --git a/src/history.cpp b/src/history.cpp index 5b748c8ce..a293933b9 100644 --- a/src/history.cpp +++ b/src/history.cpp @@ -39,7 +39,6 @@ #include "parse_util.h" #include "path.h" #include "reader.h" -#include "signal.h" #include "wutil.h" // IWYU pragma: keep // Our history format is intended to be valid YAML. Here it is: @@ -998,9 +997,6 @@ bool history_t::load_old_if_needed(void) { if (loaded_old) return true; loaded_old = true; - // PCA not sure why signals were blocked here - // signal_block(); - bool ok = false; if (map_file(name, &mmap_start, &mmap_length, &mmap_file_id)) { // Here we've mapped the file. @@ -1009,7 +1005,6 @@ bool history_t::load_old_if_needed(void) { this->populate_from_mmap(); } - // signal_unblock(); return ok; } @@ -1409,8 +1404,6 @@ bool history_t::save_internal_via_appending() { return true; } - signal_block(); - // We are going to open the file, lock it, append to it, and then close it // After locking it, we need to stat the file at the path; if there is a new file there, it // means @@ -1498,8 +1491,6 @@ bool history_t::save_internal_via_appending() { close(history_fd); } - signal_unblock(); - // If someone has replaced the file, forget our file state. if (file_changed) { this->clear_file_state(); diff --git a/src/postfork.cpp b/src/postfork.cpp index 9f29963b0..8dc49b298 100644 --- a/src/postfork.cpp +++ b/src/postfork.cpp @@ -109,9 +109,9 @@ bool set_child_group(job_t *j, process_t *p, int print_errors) { int result = -1; errno = EINTR; while (result == -1 && errno == EINTR) { - signal_block(true); + signal_block(); result = tcsetpgrp(STDIN_FILENO, j->pgid); - signal_unblock(true); + signal_unblock(); } if (result == -1) { if (errno == ENOTTY) redirect_tty_output(); @@ -251,8 +251,6 @@ int setup_child_process(job_t *j, process_t *p, const io_chain_t &io_chain) { signal_reset_handlers(); } - signal_unblock(); // remove all signal blocks - return ok ? 0 : -1; } diff --git a/src/proc.cpp b/src/proc.cpp index 063f01feb..cdbea292b 100644 --- a/src/proc.cpp +++ b/src/proc.cpp @@ -789,7 +789,7 @@ static bool terminal_give_to_job(job_t *j, int cont) { return true; } - signal_block(true); + signal_block(); int result = -1; errno = EINTR; while (result == -1 && errno == EINTR) { @@ -799,7 +799,7 @@ static bool terminal_give_to_job(job_t *j, int cont) { if (errno == ENOTTY) redirect_tty_output(); debug(1, _(L"Could not send job %d ('%ls') to foreground"), j->job_id, j->command_wcstr()); wperror(L"tcsetpgrp"); - signal_unblock(true); + signal_unblock(); return false; } @@ -817,12 +817,12 @@ static bool terminal_give_to_job(job_t *j, int cont) { debug(1, _(L"terminal_give_to_job(): Could not send job %d ('%ls') to foreground"), j->job_id, j->command_wcstr()); wperror(L"tcsetattr"); - signal_unblock(true); + signal_unblock(); return false; } } - signal_unblock(true); + signal_unblock(); return true; } @@ -835,12 +835,12 @@ static bool terminal_return_from_job(job_t *j) { return true; } - signal_block(true); + signal_block(); if (tcsetpgrp(STDIN_FILENO, getpgrp()) == -1) { if (errno == ENOTTY) redirect_tty_output(); debug(1, _(L"Could not return shell to foreground")); wperror(L"tcsetpgrp"); - signal_unblock(true); + signal_unblock(); return false; } @@ -849,7 +849,7 @@ static bool terminal_return_from_job(job_t *j) { if (errno == EIO) redirect_tty_output(); debug(1, _(L"Could not return shell to foreground")); wperror(L"tcgetattr"); - signal_unblock(true); + signal_unblock(); return false; } @@ -867,7 +867,7 @@ static bool terminal_return_from_job(job_t *j) { } #endif - signal_unblock(true); + signal_unblock(); return true; } diff --git a/src/reader.cpp b/src/reader.cpp index c7d3d8afd..7daccce2a 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -1575,20 +1575,12 @@ static void reader_interactive_init() { // Check if we are in control of the terminal, so that we don't do semi-expensive things like // reset signal handlers unless we really have to, which we often don't. if (tcgetpgrp(STDIN_FILENO) != shell_pgid) { - int block_count = 0; - int i; - // Bummer, we are not in control of the terminal. Stop until parent has given us control of - // it. Stopping in fish is a bit of a challange, what with all the signal fidgeting, we need - // to reset a bunch of signal state, making this coda a but unobvious. + // it. // // In theory, reseting signal handlers could cause us to miss signal deliveries. In - // practice, this code should only be run suring startup, when we're not waiting for any + // practice, this code should only be run during startup, when we're not waiting for any // signals. - while (signal_is_blocked()) { - signal_unblock(); - block_count++; - } signal_reset_handlers(); // Ok, signal handlers are taken out of the picture. Stop ourself in a loop until we are in @@ -1632,10 +1624,6 @@ static void reader_interactive_init() { } signal_set_handlers(); - - for (i = 0; i < block_count; i++) { - signal_block(); - } } // Put ourselves in our own process group. diff --git a/src/signal.cpp b/src/signal.cpp index 1b937825c..f36847b59 100644 --- a/src/signal.cpp +++ b/src/signal.cpp @@ -16,9 +16,6 @@ #include "reader.h" #include "wutil.h" // IWYU pragma: keep -// This is a temporary var while we explore whether signal_block() and friends is needed. -bool ignore_signal_block = true; - /// Struct describing an entry for the lookup table used to convert between signal names and signal /// ids, etc. struct lookup_entry { @@ -383,9 +380,7 @@ void get_signals_with_handlers(sigset_t *set) { } } -void signal_block(bool force) { - if (!force && ignore_signal_block) return; - +void signal_block() { ASSERT_IS_MAIN_THREAD(); sigset_t chldset; @@ -405,14 +400,10 @@ void signal_unblock_all() { sigprocmask(SIG_SETMASK, &iset, NULL); } -void signal_unblock(bool force) { - if (!force && ignore_signal_block) return; - +void signal_unblock() { ASSERT_IS_MAIN_THREAD(); - sigset_t chldset; block_count--; - if (block_count < 0) { debug(0, _(L"Signal block mismatch")); bugreport(); @@ -420,13 +411,12 @@ void signal_unblock(bool force) { } if (!block_count) { + sigset_t chldset; sigfillset(&chldset); DIE_ON_FAILURE(pthread_sigmask(SIG_UNBLOCK, &chldset, 0)); } - // debug( 0, L"signal block level decreased to %d", block_count ); } bool signal_is_blocked() { - if (ignore_signal_block) return false; return static_cast(block_count); } diff --git a/src/signal.h b/src/signal.h index 06fe37cf9..dcbc3be75 100644 --- a/src/signal.h +++ b/src/signal.h @@ -30,16 +30,14 @@ void signal_handle(int sig, int do_handle); void signal_unblock_all(); /// Block all signals. -void signal_block(bool force = false); +void signal_block(); /// Unblock all signals. -void signal_unblock(bool force = false); +void signal_unblock(); /// Returns true if signals are being blocked. bool signal_is_blocked(); /// Returns signals with non-default handlers. void get_signals_with_handlers(sigset_t *set); - -extern bool ignore_signal_block; #endif From 7876bc6ef37bcaf65fa2b8efc7500fcecf269b70 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:30:54 -0700 Subject: [PATCH 72/79] Revert "make `tokenize_variable_array()` private" This reverts commit 4fe9d794382b6b81967e6ee88c76403bb8a69541. It was meant for the major branch. --- src/env.cpp | 3 +-- src/env.h | 6 ++++++ src/expand.cpp | 4 ++-- src/expand.h | 5 ++--- src/function.cpp | 7 ++++--- src/highlight.cpp | 8 ++++---- src/output.cpp | 6 +++--- src/output.h | 2 +- src/path.cpp | 4 ++-- 9 files changed, 25 insertions(+), 20 deletions(-) diff --git a/src/env.cpp b/src/env.cpp index f3db6495a..7a6ef56d5 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -105,7 +105,6 @@ static const wcstring_list_t colon_delimited_variable({L"PATH", L"MANPATH", L"CD // Some forward declarations to make it easy to logically group the code. static void init_locale(); static void init_curses(); -static void tokenize_variable_array(const wcstring &val, wcstring_list_t &out); // Struct representing one level in the function variable stack. // Only our variable stack should create and destroy these @@ -1622,7 +1621,7 @@ std::unique_ptr list_to_array_val(const wchar_t **list) { return val; } -static void tokenize_variable_array(const wcstring &val, wcstring_list_t &out) { +void tokenize_variable_array(const wcstring &val, wcstring_list_t &out) { out.clear(); // ensure the output var is empty -- this will normally be a no-op // Zero element arrays are internally encoded as this placeholder string. diff --git a/src/env.h b/src/env.h index b7f3c2158..64a2327df 100644 --- a/src/env.h +++ b/src/env.h @@ -69,6 +69,12 @@ void env_init(const struct config_paths_t *paths = NULL); /// routines. void misc_init(); +/// Tokenize the specified string into the specified wcstring_list_t. +/// +/// \param val the input string. The contents of this string is not changed. +/// \param out the list in which to place the elements. +void tokenize_variable_array(const wcstring &val, wcstring_list_t &out); + class env_var_t { private: bool is_missing; diff --git a/src/expand.cpp b/src/expand.cpp index f265485d4..aa972a458 100644 --- a/src/expand.cpp +++ b/src/expand.cpp @@ -168,11 +168,11 @@ static int is_quotable(const wchar_t *str) { static int is_quotable(const wcstring &str) { return is_quotable(str.c_str()); } -wcstring expand_escape_variable(const env_var_t &var) { +wcstring expand_escape_variable(const wcstring &in) { wcstring_list_t lst; wcstring buff; - var.to_list(lst); + tokenize_variable_array(in, lst); size_t size = lst.size(); if (size == 0) { diff --git a/src/expand.h b/src/expand.h index 2fd0423bd..10fc88df5 100644 --- a/src/expand.h +++ b/src/expand.h @@ -13,7 +13,6 @@ #include #include "common.h" -#include "env.h" #include "parse_constants.h" enum { @@ -121,10 +120,10 @@ __warn_unused expand_error_t expand_string(const wcstring &input, std::vector &names, int get_hidden) { size_t i; - const env_var_t path_var = env_get(L"fish_function_path"); - if (path_var.missing_or_empty()) return; + const env_var_t path_var_wstr = env_get(L"fish_function_path"); + if (path_var_wstr.missing()) return; + const wchar_t *path_var = path_var_wstr.c_str(); wcstring_list_t path_list; - path_var.to_list(path_list); + tokenize_variable_array(path_var, path_list); for (i = 0; i < path_list.size(); i++) { const wcstring &ndir_str = path_list.at(i); const wchar_t *ndir = (wchar_t *)ndir_str.c_str(); diff --git a/src/highlight.cpp b/src/highlight.cpp index 2a64336f9..69c9d0a45 100644 --- a/src/highlight.cpp +++ b/src/highlight.cpp @@ -274,14 +274,14 @@ rgb_color_t highlight_get_color(highlight_spec_t highlight, bool is_background) if (var.missing()) var = env_get(highlight_var[0]); - if (!var.missing()) result = parse_color(var, treat_as_background); + if (!var.missing()) result = parse_color(var.as_string(), treat_as_background); // Handle modifiers. if (highlight & highlight_modifier_valid_path) { - env_var_t var2 = env_get(L"fish_color_valid_path"); - const wcstring val2 = var2.missing() ? L"" : var2.c_str(); + env_var_t val2_wstr = env_get(L"fish_color_valid_path"); + const wcstring val2 = val2_wstr.missing() ? L"" : val2_wstr.c_str(); - rgb_color_t result2 = parse_color(var2, is_background); + rgb_color_t result2 = parse_color(val2, is_background); if (result.is_normal()) result = result2; else { diff --git a/src/output.cpp b/src/output.cpp index 9ec4a29cd..94252223b 100644 --- a/src/output.cpp +++ b/src/output.cpp @@ -486,8 +486,8 @@ rgb_color_t best_color(const std::vector &candidates, color_support } /// Return the internal color code representing the specified color. -/// TODO: This code should be refactored to enable sharing with builtin_set_color. -rgb_color_t parse_color(const env_var_t &var, bool is_background) { +/// XXX This code should be refactored to enable sharing with builtin_set_color. +rgb_color_t parse_color(const wcstring &val, bool is_background) { int is_bold = 0; int is_underline = 0; int is_italics = 0; @@ -497,7 +497,7 @@ rgb_color_t parse_color(const env_var_t &var, bool is_background) { std::vector candidates; wcstring_list_t el; - var.to_list(el); + tokenize_variable_array(val, el); for (size_t j = 0; j < el.size(); j++) { const wcstring &next = el.at(j); diff --git a/src/output.h b/src/output.h index 58a871450..26fbdc260 100644 --- a/src/output.h +++ b/src/output.h @@ -36,7 +36,7 @@ int writech(wint_t ch); void writestr(const wchar_t *str); -rgb_color_t parse_color(const env_var_t &val, bool is_background); +rgb_color_t parse_color(const wcstring &val, bool is_background); int writeb(tputs_arg_t b); diff --git a/src/path.cpp b/src/path.cpp index 40ecd0757..1b20d8fc6 100644 --- a/src/path.cpp +++ b/src/path.cpp @@ -60,8 +60,8 @@ static bool path_get_path_core(const wcstring &cmd, wcstring *out_path, bin_path = *list_to_array_val(wcstring_list_t({L"/bin", L"/usr/bin", PREFIX L"/bin"})); } - wcstring_list_t pathsv; - bin_path_var.to_list(pathsv); + std::vector pathsv; + tokenize_variable_array(bin_path, pathsv); for (auto next_path : pathsv) { if (next_path.empty()) continue; append_path_component(next_path, cmd); From 296008089ccefd81e6dedaaebdbdee56fb9b4d21 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:31:56 -0700 Subject: [PATCH 73/79] Revert "stop subclassing env_var_t from wcstring" This reverts commit c36ad2761892a8f7dbb9cffc1ab2c2a470b93bf5. It was meant for the major branch. --- src/autoload.cpp | 6 +- src/autoload.h | 3 +- src/builtin_cd.cpp | 2 +- src/builtin_functions.cpp | 2 +- src/builtin_read.cpp | 4 +- src/builtin_set.cpp | 42 +++++----- src/common.cpp | 8 +- src/complete.cpp | 9 +-- src/env.cpp | 160 +++++++++++++++++--------------------- src/env.h | 60 +++++++------- src/exec.cpp | 4 +- src/expand.cpp | 24 +++--- src/function.cpp | 4 +- src/highlight.cpp | 12 +-- src/history.cpp | 5 +- src/input.cpp | 4 +- src/input_common.cpp | 2 +- src/output.cpp | 2 +- src/path.cpp | 24 +++--- src/path.h | 2 +- src/reader.cpp | 4 +- src/screen.cpp | 4 +- 22 files changed, 184 insertions(+), 203 deletions(-) diff --git a/src/autoload.cpp b/src/autoload.cpp index a9f944aed..ae3a7aafc 100644 --- a/src/autoload.cpp +++ b/src/autoload.cpp @@ -69,7 +69,7 @@ int autoload_t::load(const wcstring &cmd, bool reload) { CHECK_BLOCK(0); ASSERT_IS_MAIN_THREAD(); - env_var_t path_var = env_get(env_var_name); + env_var_t path_var = env_get_string(env_var_name); // Do we know where to look? if (path_var.empty()) return 0; @@ -79,7 +79,7 @@ int autoload_t::load(const wcstring &cmd, bool reload) { if (path_var != this->last_path) { this->last_path = path_var; this->last_path_tokenized.clear(); - this->last_path.to_list(this->last_path_tokenized); + tokenize_variable_array(this->last_path, this->last_path_tokenized); scoped_lock locker(lock); this->evict_all_nodes(); @@ -115,7 +115,7 @@ bool autoload_t::can_load(const wcstring &cmd, const env_vars_snapshot_t &vars) if (path_var.missing_or_empty()) return false; std::vector path_list; - path_var.to_list(path_list); + tokenize_variable_array(path_var, path_list); return this->locate_file_and_maybe_load_it(cmd, false, false, path_list); } diff --git a/src/autoload.h b/src/autoload.h index d04003c25..63886390e 100644 --- a/src/autoload.h +++ b/src/autoload.h @@ -8,7 +8,6 @@ #include #include "common.h" -#include "env.h" #include "lru.h" /// Record of an attempt to access a file. @@ -51,7 +50,7 @@ class autoload_t : public lru_cache_t { /// The environment variable name. const wcstring env_var_name; /// The path from which we most recently autoloaded. - env_var_t last_path; + wcstring last_path; /// the most reecently autoloaded path, tokenized (split on separators). wcstring_list_t last_path_tokenized; /// A table containing all the files that are currently being loaded. diff --git a/src/builtin_cd.cpp b/src/builtin_cd.cpp index 8513f4a50..e73808fab 100644 --- a/src/builtin_cd.cpp +++ b/src/builtin_cd.cpp @@ -39,7 +39,7 @@ int builtin_cd(parser_t &parser, io_streams_t &streams, wchar_t **argv) { if (argv[optind]) { dir_in = env_var_t(argv[optind]); } else { - dir_in = env_get(L"HOME"); + dir_in = env_get_string(L"HOME"); if (dir_in.missing_or_empty()) { streams.err.append_format(_(L"%ls: Could not find home directory\n"), cmd); return STATUS_CMD_ERROR; diff --git a/src/builtin_functions.cpp b/src/builtin_functions.cpp index 6a3a3ef49..5e6b42433 100644 --- a/src/builtin_functions.cpp +++ b/src/builtin_functions.cpp @@ -193,7 +193,7 @@ static wcstring functions_def(const wcstring &name) { it != end; ++it) { wcstring_list_t lst; if (!it->second.missing()) { - it->second.to_list(lst); + tokenize_variable_array(it->second, lst); } // This forced tab is crummy, but we don't know what indentation style the function uses. diff --git a/src/builtin_read.cpp b/src/builtin_read.cpp index cfd32ace2..0291a084a 100644 --- a/src/builtin_read.cpp +++ b/src/builtin_read.cpp @@ -427,9 +427,9 @@ int builtin_read(parser_t &parser, io_streams_t &streams, wchar_t **argv) { } if (!opts.have_delimiter) { - env_var_t ifs = env_get(L"IFS"); + env_var_t ifs = env_get_string(L"IFS"); if (!ifs.missing_or_empty()) { - opts.delimiter = ifs.as_string(); + opts.delimiter = ifs; } } if (opts.delimiter.empty()) { diff --git a/src/builtin_set.cpp b/src/builtin_set.cpp index cad1034ae..daa713a99 100644 --- a/src/builtin_set.cpp +++ b/src/builtin_set.cpp @@ -239,8 +239,9 @@ static int my_env_path_setup(const wchar_t *cmd, const wchar_t *key, //!OCLINT( // not the (missing) local value. Also don't bother to complain about relative paths, which // don't start with /. wcstring_list_t existing_values; - const env_var_t existing_variable = env_get(key, ENV_DEFAULT); - if (!existing_variable.missing_or_empty()) existing_variable.to_list(existing_values); + const env_var_t existing_variable = env_get_string(key, ENV_DEFAULT); + if (!existing_variable.missing_or_empty()) + tokenize_variable_array(existing_variable, existing_values); for (size_t i = 0; i < list.size(); i++) { const wcstring &dir = list.at(i); @@ -345,9 +346,9 @@ static int parse_index(std::vector &indexes, wchar_t *src, int scope, io_s *p = L'\0'; // split the var name from the indexes/slices p++; - env_var_t var_str = env_get(src, scope); + env_var_t var_str = env_get_string(src, scope); wcstring_list_t var; - if (!var_str.missing()) var_str.to_list(var); + if (!var_str.missing()) tokenize_variable_array(var_str, var); int count = 0; @@ -456,16 +457,15 @@ static int builtin_set_list(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, streams.out.append(e_key); if (!names_only) { - env_var_t var = env_get(key, compute_scope(opts)); - if (!var.missing()) { + env_var_t value = env_get_string(key, compute_scope(opts)); + if (!value.missing()) { bool shorten = false; - wcstring val = var.as_string(); - if (opts.shorten_ok && val.length() > 64) { + if (opts.shorten_ok && value.length() > 64) { shorten = true; - val.resize(60); + value.resize(60); } - wcstring e_value = expand_escape_variable(val); + wcstring e_value = expand_escape_variable(value); streams.out.append(L" "); streams.out.append(e_value); @@ -500,8 +500,8 @@ static int builtin_set_query(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, if (idx_count) { wcstring_list_t result; - env_var_t dest_str = env_get(dest, scope); - if (!dest_str.missing()) dest_str.to_list(result); + env_var_t dest_str = env_get_string(dest, scope); + if (!dest_str.missing()) tokenize_variable_array(dest_str, result); for (auto idx : indexes) { if (idx < 1 || (size_t)idx > result.size()) retval++; @@ -538,12 +538,12 @@ static void show_scope(const wchar_t *var_name, int scope, io_streams_t &streams } if (env_exist(var_name, scope)) { - const env_var_t evar = env_get(var_name, scope | ENV_EXPORT | ENV_USER); + const env_var_t evar = env_get_string(var_name, scope | ENV_EXPORT | ENV_USER); const wchar_t *exportv = evar.missing() ? _(L"unexported") : _(L"exported"); - const env_var_t var = env_get(var_name, scope | ENV_USER); + const env_var_t var = env_get_string(var_name, scope | ENV_USER); wcstring_list_t result; - if (!var.empty()) var.to_list(result); + if (!var.empty()) tokenize_variable_array(var, result); streams.out.append_format(_(L"$%ls: set in %ls scope, %ls, with %d elements\n"), var_name, scope_name, exportv, result.size()); @@ -632,10 +632,10 @@ static int builtin_set_erase(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, if (idx_count == 0) { // unset the var retval = env_remove(dest, scope); } else { // remove just the specified indexes of the var - const env_var_t dest_var = env_get(dest, scope); + const env_var_t dest_var = env_get_string(dest, scope); if (dest_var.missing()) return STATUS_CMD_ERROR; wcstring_list_t result; - dest_var.to_list(result); + tokenize_variable_array(dest_var, result); erase_values(result, indexes); retval = my_env_set(cmd, dest, result, scope, streams); } @@ -658,9 +658,9 @@ static int set_var_array(const wchar_t *cmd, set_cmd_opts_t &opts, const wchar_t for (int i = 0; i < argc; i++) new_values.push_back(argv[i]); } - env_var_t var_str = env_get(varname, scope); + env_var_t var_str = env_get_string(varname, scope); wcstring_list_t var_array; - if (!var_str.missing()) var_str.to_list(var_array); + if (!var_str.missing()) tokenize_variable_array(var_str, var_array); new_values.insert(new_values.end(), var_array.begin(), var_array.end()); if (opts.append) { @@ -691,8 +691,8 @@ static int set_var_slices(const wchar_t *cmd, set_cmd_opts_t &opts, const wchar_ } int scope = compute_scope(opts); // calculate the variable scope based on the provided options - const env_var_t var_str = env_get(varname, scope); - if (!var_str.missing()) var_str.to_list(new_values); + const env_var_t var_str = env_get_string(varname, scope); + if (!var_str.missing()) tokenize_variable_array(var_str, new_values); // Slice indexes have been calculated, do the actual work. wcstring_list_t result; diff --git a/src/common.cpp b/src/common.cpp index 1dfbdda86..be5c6bf43 100644 --- a/src/common.cpp +++ b/src/common.cpp @@ -1554,8 +1554,8 @@ static void validate_new_termsize(struct winsize *new_termsize) { } #endif // Fallback to the environment vars. - env_var_t col_var = env_get(L"COLUMNS"); - env_var_t row_var = env_get(L"LINES"); + env_var_t col_var = env_get_string(L"COLUMNS"); + env_var_t row_var = env_get_string(L"LINES"); if (!col_var.missing_or_empty() && !row_var.missing_or_empty()) { // Both vars have to have valid values. int col = fish_wcstoi(col_var.c_str()); @@ -1582,11 +1582,11 @@ static void validate_new_termsize(struct winsize *new_termsize) { /// Export the new terminal size as env vars and to the kernel if possible. static void export_new_termsize(struct winsize *new_termsize) { wchar_t buf[64]; - env_var_t cols = env_get(L"COLUMNS", ENV_EXPORT); + env_var_t cols = env_get_string(L"COLUMNS", ENV_EXPORT); swprintf(buf, 64, L"%d", (int)new_termsize->ws_col); env_set(L"COLUMNS", buf, ENV_GLOBAL | (cols.missing_or_empty() ? 0 : ENV_EXPORT)); - env_var_t lines = env_get(L"LINES", ENV_EXPORT); + env_var_t lines = env_get_string(L"LINES", ENV_EXPORT); swprintf(buf, 64, L"%d", (int)new_termsize->ws_row); env_set(L"LINES", buf, ENV_GLOBAL | (lines.missing_or_empty() ? 0 : ENV_EXPORT)); diff --git a/src/complete.cpp b/src/complete.cpp index 2df3cb501..51b6acd91 100644 --- a/src/complete.cpp +++ b/src/complete.cpp @@ -1106,13 +1106,12 @@ bool completer_t::complete_variable(const wcstring &str, size_t start_offset) { wcstring desc; if (this->wants_descriptions()) { // Can't use this->vars here, it could be any variable. - env_var_t var = env_get(env_name); - if (var.missing()) continue; + env_var_t value_unescaped = env_get_string(env_name); + if (value_unescaped.missing()) continue; - wcstring value = expand_escape_variable(var.as_string()); - if (this->type() != COMPLETE_AUTOSUGGEST) { + wcstring value = expand_escape_variable(value_unescaped); + if (this->type() != COMPLETE_AUTOSUGGEST) desc = format_string(COMPLETE_VAR_DESC_VAL, value.c_str()); - } } append_completion(&this->completions, comp, desc, flags, match); diff --git a/src/env.cpp b/src/env.cpp index 7a6ef56d5..0fd675539 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -331,13 +331,13 @@ static bool var_is_timezone(const wcstring &key) { return key == L"TZ"; } /// Properly sets all timezone information. static void handle_timezone(const wchar_t *env_var_name) { debug(2, L"handle_timezone() called in response to '%ls' changing", env_var_name); - const env_var_t var = env_get(env_var_name, ENV_EXPORT); + const env_var_t val = env_get_string(env_var_name, ENV_EXPORT); + const std::string &value = wcs2string(val); const std::string &name = wcs2string(env_var_name); - debug(2, L"timezone var %s='%s'", name.c_str(), var.c_str()); - if (var.missing_or_empty()) { + debug(2, L"timezone var %s='%s'", name.c_str(), value.c_str()); + if (val.empty()) { unsetenv(name.c_str()); } else { - const std::string &value = wcs2string(var.as_string()); setenv(name.c_str(), value.c_str(), 1); } tzset(); @@ -350,13 +350,13 @@ static void fix_colon_delimited_var(const wcstring &var_name) { // While we auto split/join MANPATH we do not want to replace empty elements with "." (#4158). if (var_name == L"MANPATH") return; - const env_var_t paths = env_get(var_name); + const env_var_t paths = env_get_string(var_name); if (paths.missing_or_empty()) return; bool modified = false; wcstring_list_t pathsv; wcstring_list_t new_pathsv; - paths.to_list(pathsv); + tokenize_variable_array(paths, pathsv); for (auto next_path : pathsv) { if (next_path.empty()) { next_path = L"."; @@ -400,14 +400,13 @@ static void init_locale() { char *old_msg_locale = strdup(setlocale(LC_MESSAGES, NULL)); for (auto var_name : locale_variables) { - const env_var_t var = env_get(var_name, ENV_EXPORT); + const env_var_t val = env_get_string(var_name, ENV_EXPORT); const std::string &name = wcs2string(var_name); - if (var.missing_or_empty()) { - debug(2, L"locale var %s missing or empty", name.c_str()); + const std::string &value = wcs2string(val); + debug(2, L"locale var %s='%s'", name.c_str(), value.c_str()); + if (val.empty()) { unsetenv(name.c_str()); } else { - const std::string &value = wcs2string(var.as_string()); - debug(2, L"locale var %s='%s'", name.c_str(), value.c_str()); setenv(name.c_str(), value.c_str(), 1); } } @@ -456,17 +455,17 @@ bool term_supports_setting_title() { return can_set_term_title; } /// don't. Since we can't see the underlying terminal below screen there is no way to fix this. static const wcstring_list_t title_terms({L"xterm", L"screen", L"tmux", L"nxterm", L"rxvt"}); static bool does_term_support_setting_title() { - const env_var_t term_var = env_get(L"TERM"); - if (term_var.missing_or_empty()) return false; + const env_var_t term_str = env_get_string(L"TERM"); + if (term_str.missing()) return false; - const wchar_t *term = term_var.c_str(); - bool recognized = contains(title_terms, term_var.as_string()); + const wchar_t *term = term_str.c_str(); + bool recognized = contains(title_terms, term_str); if (!recognized) recognized = !wcsncmp(term, L"xterm-", wcslen(L"xterm-")); if (!recognized) recognized = !wcsncmp(term, L"screen-", wcslen(L"screen-")); if (!recognized) recognized = !wcsncmp(term, L"tmux-", wcslen(L"tmux-")); if (!recognized) { - if (wcscmp(term, L"linux") == 0) return false; - if (wcscmp(term, L"dumb") == 0) return false; + if (term_str == L"linux") return false; + if (term_str == L"dumb") return false; char buf[PATH_MAX]; int retval = ttyname_r(STDIN_FILENO, buf, PATH_MAX); @@ -480,12 +479,11 @@ static bool does_term_support_setting_title() { static void update_fish_color_support() { // Detect or infer term256 support. If fish_term256 is set, we respect it; // otherwise infer it from the TERM variable or use terminfo. - env_var_t fish_term256 = env_get(L"fish_term256"); - env_var_t term_var = env_get(L"TERM"); - wcstring term = term_var.missing_or_empty() ? L"" : term_var.as_string(); + env_var_t fish_term256 = env_get_string(L"fish_term256"); + env_var_t term = env_get_string(L"TERM"); bool support_term256 = false; // default to no support if (!fish_term256.missing_or_empty()) { - support_term256 = from_string(fish_term256.as_string()); + support_term256 = from_string(fish_term256); debug(2, L"256 color support determined by 'fish_term256'"); } else if (term.find(L"256color") != wcstring::npos) { // TERM=*256color*: Explicitly supported. @@ -493,12 +491,11 @@ static void update_fish_color_support() { debug(2, L"256 color support enabled for '256color' in TERM"); } else if (term.find(L"xterm") != wcstring::npos) { // Assume that all xterms are 256, except for OS X SnowLeopard - const env_var_t prog_var = env_get(L"TERM_PROGRAM"); - const env_var_t progver_var = env_get(L"TERM_PROGRAM_VERSION"); - wcstring term_program = prog_var.missing_or_empty() ? L"" : prog_var.as_string(); - if (term_program == L"Apple_Terminal" && !progver_var.missing_or_empty()) { + const env_var_t prog = env_get_string(L"TERM_PROGRAM"); + const env_var_t progver = env_get_string(L"TERM_PROGRAM_VERSION"); + if (prog == L"Apple_Terminal" && !progver.missing_or_empty()) { // OS X Lion is version 300+, it has 256 color support - if (strtod(wcs2str(progver_var.as_string()), NULL) > 300) { + if (strtod(wcs2str(progver), NULL) > 300) { support_term256 = true; debug(2, L"256 color support enabled for TERM=xterm + modern Terminal.app"); } @@ -514,10 +511,10 @@ static void update_fish_color_support() { debug(2, L"256 color support not enabled (yet)"); } - env_var_t fish_term24bit = env_get(L"fish_term24bit"); + env_var_t fish_term24bit = env_get_string(L"fish_term24bit"); bool support_term24bit; if (!fish_term24bit.missing_or_empty()) { - support_term24bit = from_string(fish_term24bit.as_string()); + support_term24bit = from_string(fish_term24bit); debug(2, L"'fish_term24bit' preference: 24-bit color %s", support_term24bit ? L"enabled" : L"disabled"); } else { @@ -537,14 +534,12 @@ static void update_fish_color_support() { static bool initialize_curses_using_fallback(const char *term) { // If $TERM is already set to the fallback name we're about to use there isn't any point in // seeing if the fallback name can be used. - env_var_t term_var = env_get(L"TERM"); - if (term_var.missing_or_empty()) return false; - - const char *term_env = wcs2str(term_var.as_string()); + const char *term_env = wcs2str(env_get_string(L"TERM")); if (!strcmp(term_env, DEFAULT_TERM1) || !strcmp(term_env, DEFAULT_TERM2)) return false; - if (is_interactive_session) debug(1, _(L"Using fallback terminal type '%s'."), term); - + if (is_interactive_session) { + debug(1, _(L"Using fallback terminal type '%s'."), term); + } int err_ret; if (setupterm((char *)term, STDOUT_FILENO, &err_ret) == OK) return true; if (is_interactive_session) { @@ -564,21 +559,20 @@ static void init_path_vars() { /// Initialize the curses subsystem. static void init_curses() { for (auto var_name : curses_variables) { - const env_var_t var = env_get(var_name, ENV_EXPORT); + const env_var_t val = env_get_string(var_name, ENV_EXPORT); const std::string &name = wcs2string(var_name); - if (var.missing_or_empty()) { - debug(2, L"curses var %s missing or empty", name.c_str()); + const std::string &value = wcs2string(val); + debug(2, L"curses var %s='%s'", name.c_str(), value.c_str()); + if (val.empty()) { unsetenv(name.c_str()); } else { - const std::string &value = wcs2string(var.as_string()); - debug(2, L"curses var %s='%s'", name.c_str(), value.c_str()); setenv(name.c_str(), value.c_str(), 1); } } int err_ret; if (setupterm(NULL, STDOUT_FILENO, &err_ret) == ERR) { - env_var_t term = env_get(L"TERM"); + env_var_t term = env_get_string(L"TERM"); if (is_interactive_session) { debug(1, _(L"Could not set up terminal.")); if (term.missing_or_empty()) { @@ -672,7 +666,7 @@ static void universal_callback(fish_message_type_t type, const wchar_t *name) { /// Make sure the PATH variable contains something. static void setup_path() { - const env_var_t path = env_get(L"PATH"); + const env_var_t path = env_get_string(L"PATH"); if (path.missing_or_empty()) { const wchar_t *value = L"/usr/bin" ARRAY_SEP_STR L"/bin"; env_set(L"PATH", value, ENV_GLOBAL | ENV_EXPORT); @@ -683,10 +677,10 @@ static void setup_path() { /// defaults. They will be updated later by the `get_current_winsize()` function if they need to be /// adjusted. static void env_set_termsize() { - env_var_t cols = env_get(L"COLUMNS"); + env_var_t cols = env_get_string(L"COLUMNS"); if (cols.missing_or_empty()) env_set(L"COLUMNS", DFLT_TERM_COL_STR, ENV_GLOBAL); - env_var_t rows = env_get(L"LINES"); + env_var_t rows = env_get_string(L"LINES"); if (rows.missing_or_empty()) env_set(L"LINES", DFLT_TERM_ROW_STR, ENV_GLOBAL); } @@ -704,7 +698,7 @@ bool env_set_pwd() { /// Allow the user to override the limit on how much data the `read` command will process. /// This is primarily for testing but could be used by users in special situations. void env_set_read_limit() { - env_var_t read_byte_limit_var = env_get(L"FISH_READ_BYTE_LIMIT"); + env_var_t read_byte_limit_var = env_get_string(L"FISH_READ_BYTE_LIMIT"); if (!read_byte_limit_var.missing_or_empty()) { size_t limit = fish_wcstoull(read_byte_limit_var.c_str()); if (errno) { @@ -716,11 +710,10 @@ void env_set_read_limit() { } wcstring env_get_pwd_slash(void) { - env_var_t pwd_var = env_get(L"PWD"); - if (pwd_var.missing_or_empty()) { + env_var_t pwd = env_get_string(L"PWD"); + if (pwd.missing_or_empty()) { return L""; } - wcstring pwd = pwd_var.as_string(); if (!string_suffixes_string(L"/", pwd)) { pwd.push_back(L'/'); } @@ -729,7 +722,7 @@ wcstring env_get_pwd_slash(void) { /// Set up the USER variable. static void setup_user(bool force) { - if (env_get(L"USER").missing_or_empty() || force) { + if (env_get_string(L"USER").missing_or_empty() || force) { struct passwd userinfo; struct passwd *result; char buf[8192]; @@ -870,12 +863,12 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { env_set(L"FISH_VERSION", version.c_str(), ENV_GLOBAL); // Set up SHLVL variable. - const env_var_t shlvl_var = env_get(L"SHLVL"); + const env_var_t shlvl_str = env_get_string(L"SHLVL"); wcstring nshlvl_str = L"1"; - if (!shlvl_var.missing_or_empty()) { + if (!shlvl_str.missing()) { const wchar_t *end; // TODO: Figure out how to handle invalid numbers better. Shouldn't we issue a diagnostic? - long shlvl_i = fish_wcstol(shlvl_var.c_str(), &end); + long shlvl_i = fish_wcstol(shlvl_str.c_str(), &end); if (!errno && shlvl_i >= 0) { nshlvl_str = to_string(shlvl_i + 1); } @@ -888,10 +881,10 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { // if the target user is root, unless "--preserve-environment" is used. // Since that is an explicit choice, we should allow it to enable e.g. // env HOME=(mktemp -d) su --preserve-environment fish - if (env_get(L"HOME").missing_or_empty()) { - env_var_t user_var = env_get(L"USER"); - if (!user_var.missing_or_empty()) { - char *unam_narrow = wcs2str(user_var.c_str()); + if (env_get_string(L"HOME").missing_or_empty()) { + const env_var_t unam = env_get_string(L"USER"); + if (!unam.missing_or_empty()) { + char *unam_narrow = wcs2str(unam.c_str()); struct passwd userinfo; struct passwd *result; char buf[8192]; @@ -899,11 +892,9 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { if (retval || !result) { // Maybe USER is set but it's bogus. Reset USER from the db and try again. setup_user(true); - user_var = env_get(L"USER"); - if (!user_var.missing_or_empty()) { - unam_narrow = wcs2str(user_var.c_str()); - retval = getpwnam_r(unam_narrow, &userinfo, buf, sizeof(buf), &result); - } + const env_var_t unam = env_get_string(L"USER"); + unam_narrow = wcs2str(unam.c_str()); + retval = getpwnam_r(unam_narrow, &userinfo, buf, sizeof(buf), &result); } if (!retval && result && userinfo.pw_dir) { const wcstring dir = str2wcstring(userinfo.pw_dir); @@ -927,8 +918,9 @@ void env_init(const struct config_paths_t *paths /* or NULL */) { env_set_read_limit(); // initialize the read_byte_limit // Set g_use_posix_spawn. Default to true. - env_var_t use_posix_spawn = env_get(L"fish_use_posix_spawn"); - g_use_posix_spawn = use_posix_spawn.missing_or_empty() ? true : from_string(use_posix_spawn.as_string()); + env_var_t use_posix_spawn = env_get_string(L"fish_use_posix_spawn"); + g_use_posix_spawn = + (use_posix_spawn.missing_or_empty() ? true : from_string(use_posix_spawn)); // Set fish_bind_mode to "default". env_set(FISH_BIND_MODE_VAR, DEFAULT_BIND_MODE, ENV_GLOBAL); @@ -1216,22 +1208,12 @@ int env_remove(const wcstring &key, int var_mode) { return !erased; } -wcstring env_var_t::as_string(void) const { - assert(!is_missing); - return val; -} - const wchar_t *env_var_t::c_str(void) const { - assert(!is_missing); - return val.c_str(); + assert(!is_missing); //!OCLINT(multiple unary operator) + return wcstring::c_str(); } -void env_var_t::to_list(wcstring_list_t &out) const { - assert(!is_missing); - tokenize_variable_array(val, out); -} - -env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { +env_var_t env_get_string(const wcstring &key, env_mode_flags_t mode) { const bool has_scope = mode & (ENV_LOCAL | ENV_GLOBAL | ENV_UNIVERSAL); const bool search_local = !has_scope || (mode & ENV_LOCAL); const bool search_global = !has_scope || (mode & ENV_GLOBAL); @@ -1250,18 +1232,18 @@ env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { if (!is_main_thread()) { return env_var_t::missing_var(); } + env_var_t result; history_t *history = reader_get_history(); if (!history) { history = &history_t::history_with_name(history_session_id()); } - wcstring result; if (history) history->get_string_representation(&result, ARRAY_SEP_STR); - return env_var_t(result); + return result; } else if (key == L"status") { - return env_var_t(to_string(proc_get_last_status())); + return to_string(proc_get_last_status()); } else if (key == L"umask") { - return env_var_t(format_string(L"0%0.3o", get_umask())); + return format_string(L"0%0.3o", get_umask()); } // We should never get here unless the electric var list is out of sync with the above code. DIE("unerecognized electric var name"); @@ -1294,7 +1276,7 @@ env_var_t env_get(const wcstring &key, env_mode_flags_t mode) { if (!search_universal) return env_var_t::missing_var(); // Another hack. Only do a universal barrier on the main thread (since it can change variable - // values). Make sure we do this outside the env_lock because it may itself call `env_get()`. + // values). Make sure we do this outside the env_lock because it may itself call env_get_string. if (is_main_thread() && !get_proc_had_barrier()) { set_proc_had_barrier(true); env_universal_barrier(); @@ -1505,12 +1487,12 @@ void var_stack_t::update_export_array_if_necessary() { const wcstring_list_t uni = uvars()->get_names(true, false); for (size_t i = 0; i < uni.size(); i++) { const wcstring &key = uni.at(i); - const env_var_t var = uvars()->get(key); + const env_var_t val = uvars()->get(key); - if (!var.missing() && var.as_string() != ENV_NULL) { + if (!val.missing() && val != ENV_NULL) { // Note that std::map::insert does NOT overwrite a value already in the map, // which we depend on here. - vals.insert(std::pair(key, var.as_string())); + vals.insert(std::pair(key, val)); } } } @@ -1551,16 +1533,16 @@ env_vars_snapshot_t::env_vars_snapshot_t(const wchar_t *const *keys) { wcstring key; for (size_t i = 0; keys[i]; i++) { key.assign(keys[i]); - const env_var_t var = env_get(key); - if (!var.missing()) { - vars[key] = var.as_string(); + const env_var_t val = env_get_string(key); + if (!val.missing()) { + vars[key] = val; } } } env_vars_snapshot_t::env_vars_snapshot_t() {} -// The "current" variables are not a snapshot at all, but instead trampoline to env_get, etc. +// The "current" variables are not a snapshot at all, but instead trampoline to env_get_string, etc. // We identify the current snapshot based on pointer values. static const env_vars_snapshot_t sCurrentSnapshot; const env_vars_snapshot_t &env_vars_snapshot_t::current() { return sCurrentSnapshot; } @@ -1568,9 +1550,9 @@ const env_vars_snapshot_t &env_vars_snapshot_t::current() { return sCurrentSnaps bool env_vars_snapshot_t::is_current() const { return this == &sCurrentSnapshot; } env_var_t env_vars_snapshot_t::get(const wcstring &key) const { - // If we represent the current state, bounce to env_get. + // If we represent the current state, bounce to env_get_string. if (this->is_current()) { - return env_get(key); + return env_get_string(key); } std::map::const_iterator iter = vars.find(key); return iter == vars.end() ? env_var_t::missing_var() : env_var_t(iter->second); diff --git a/src/env.h b/src/env.h index 64a2327df..7160e893d 100644 --- a/src/env.h +++ b/src/env.h @@ -24,7 +24,7 @@ extern bool curses_initialized; /// Value denoting a null string. #define ENV_NULL L"\x1d" -// Flags that may be passed as the 'mode' in env_set / env_get. +// Flags that may be passed as the 'mode' in env_set / env_get_string. enum { /// Default mode. ENV_DEFAULT = 0, @@ -69,16 +69,11 @@ void env_init(const struct config_paths_t *paths = NULL); /// routines. void misc_init(); -/// Tokenize the specified string into the specified wcstring_list_t. -/// -/// \param val the input string. The contents of this string is not changed. -/// \param out the list in which to place the elements. -void tokenize_variable_array(const wcstring &val, wcstring_list_t &out); +int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t mode); -class env_var_t { +class env_var_t : public wcstring { private: bool is_missing; - wcstring val; public: static env_var_t missing_var() { @@ -87,36 +82,41 @@ class env_var_t { return result; } - env_var_t(const env_var_t &x) : val(x.val), is_missing(x.is_missing) {} - env_var_t(const wcstring &x) : val(x), is_missing(false) {} - env_var_t(const wchar_t *x) : val(x), is_missing(false) {} - env_var_t() : val(L""), is_missing(false) {} + env_var_t(const env_var_t &x) : wcstring(x), is_missing(x.is_missing) {} + env_var_t(const wcstring &x) : wcstring(x), is_missing(false) {} + env_var_t(const wchar_t *x) : wcstring(x), is_missing(false) {} + env_var_t() : wcstring(L""), is_missing(false) {} - bool empty(void) const { return val.empty(); }; bool missing(void) const { return is_missing; } - bool missing_or_empty(void) const { return missing() || val.empty(); } + + bool missing_or_empty(void) const { return missing() || empty(); } const wchar_t *c_str(void) const; - void to_list(wcstring_list_t &out) const; - wcstring as_string() const; - env_var_t &operator=(const env_var_t &v) { - is_missing = v.is_missing; - val = v.val; + env_var_t &operator=(const env_var_t &s) { + is_missing = s.is_missing; + wcstring::operator=(s); return *this; } - bool operator==(const env_var_t &s) const { return is_missing == s.is_missing && val == s.val; } + bool operator==(const env_var_t &s) const { + return is_missing == s.is_missing && + static_cast(*this) == static_cast(s); + } - bool operator==(const wcstring &s) const { return !is_missing && val == s; } + bool operator==(const wcstring &s) const { + return !is_missing && static_cast(*this) == s; + } - bool operator!=(const env_var_t &v) const { return val != v.val; } + bool operator!=(const env_var_t &s) const { return !(*this == s); } - bool operator!=(const wcstring &s) const { return val != s; } + bool operator!=(const wcstring &s) const { return !(*this == s); } - bool operator!=(const wchar_t *s) const { return val != s; } + bool operator==(const wchar_t *s) const { + return !is_missing && static_cast(*this) == s; + } - bool operator==(const wchar_t *s) const { return !is_missing && val == s; } + bool operator!=(const wchar_t *s) const { return !(*this == s); } }; /// Gets the variable with the specified name, or env_var_t::missing_var if it does not exist or is @@ -124,9 +124,7 @@ class env_var_t { /// /// \param key The name of the variable to get /// \param mode An optional scope to search in. All scopes are searched if unset -env_var_t env_get(const wcstring &key, env_mode_flags_t mode = ENV_DEFAULT); - -int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t mode); +env_var_t env_get_string(const wcstring &key, env_mode_flags_t mode = ENV_DEFAULT); /// Returns true if the specified key exists. This can't be reliably done using env_get, since /// env_get returns null for 0-element arrays. @@ -216,4 +214,10 @@ bool term_supports_setting_title(); /// Returns the fish internal representation for an array of strings. std::unique_ptr list_to_array_val(const wcstring_list_t &list); std::unique_ptr list_to_array_val(const wchar_t **list); + +/// Tokenize the specified string into the specified wcstring_list_t. +/// +/// \param val the input string. The contents of this string is not changed. +/// \param out the list in which to place the elements. +void tokenize_variable_array(const wcstring &val, wcstring_list_t &out); #endif diff --git a/src/exec.cpp b/src/exec.cpp index 5af251572..44f41255c 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -405,7 +405,7 @@ void exec_job(parser_t &parser, job_t *j) { // really make sense, so I'm not trying to fix it here. if (!setup_child_process(j, 0, all_ios)) { // Decrement SHLVL as we're removing ourselves from the shell "stack". - const env_var_t shlvl_str = env_get(L"SHLVL", ENV_GLOBAL | ENV_EXPORT); + const env_var_t shlvl_str = env_get_string(L"SHLVL", ENV_GLOBAL | ENV_EXPORT); wcstring nshlvl_str = L"0"; if (!shlvl_str.missing()) { long shlvl_i = fish_wcstol(shlvl_str.c_str()); @@ -1111,7 +1111,7 @@ static int exec_subshell_internal(const wcstring &cmd, wcstring_list_t *lst, boo const int prev_status = proc_get_last_status(); bool split_output = false; - const env_var_t ifs = env_get(L"IFS"); + const env_var_t ifs = env_get_string(L"IFS"); if (!ifs.missing_or_empty()) { split_output = true; } diff --git a/src/expand.cpp b/src/expand.cpp index aa972a458..79f4cec7d 100644 --- a/src/expand.cpp +++ b/src/expand.cpp @@ -144,7 +144,7 @@ static void append_cmdsub_error(parse_error_list_t *errors, size_t source_start, /// Return the environment variable value for the string starting at \c in. static env_var_t expand_var(const wchar_t *in) { if (!in) return env_var_t::missing_var(); - return env_get(in); + return env_get_string(in); } /// Test if the specified string does not contain character which can not be used inside a quoted @@ -777,19 +777,19 @@ static int expand_variables(const wcstring &instr, std::vector *ou } var_tmp.append(instr, start_pos, var_len); - env_var_t var; + env_var_t var_val; if (var_len == 1 && var_tmp[0] == VARIABLE_EXPAND_EMPTY) { - var = env_var_t::missing_var(); + var_val = env_var_t::missing_var(); } else { - var = expand_var(var_tmp.c_str()); + var_val = expand_var(var_tmp.c_str()); } - if (!var.missing()) { + if (!var_val.missing()) { int all_vars = 1; wcstring_list_t var_item_list; if (is_ok) { - var.to_list(var_item_list); + tokenize_variable_array(var_val, var_item_list); const size_t slice_start = stop_pos; if (slice_start < insize && instr.at(slice_start) == L'[') { @@ -1173,7 +1173,7 @@ static void expand_home_directory(wcstring &input) { env_var_t home; if (username.empty()) { // Current users home directory. - home = env_get(L"HOME"); + home = env_get_string(L"HOME"); // If home is either missing or empty, // treat it like an empty list. // $HOME is defined to be a _path_, @@ -1201,7 +1201,7 @@ static void expand_home_directory(wcstring &input) { } } - wchar_t *realhome = wrealpath(home.as_string(), NULL); + wchar_t *realhome = wrealpath(home, NULL); if (!tilde_error && realhome) { input.replace(input.begin(), input.begin() + tail_idx, realhome); @@ -1433,12 +1433,12 @@ static expand_error_t expand_stage_wildcards(const wcstring &input, std::vector< } else { // Get the PATH/CDPATH and cwd. Perhaps these should be passed in. An empty CDPATH // implies just the current directory, while an empty PATH is left empty. - env_var_t paths = env_get(for_cd ? L"CDPATH" : L"PATH"); + env_var_t paths = env_get_string(for_cd ? L"CDPATH" : L"PATH"); if (paths.missing_or_empty()) paths = for_cd ? L"." : L""; // Tokenize it into path names. std::vector pathsv; - paths.to_list(pathsv); + tokenize_variable_array(paths, pathsv); for (auto next_path : pathsv) { effective_working_dirs.push_back( path_apply_working_directory(next_path, working_dir)); @@ -1592,9 +1592,9 @@ void update_abbr_cache(const wchar_t *op, const wcstring varname) { } abbreviations.erase(abbr); if (wcscmp(op, L"ERASE") != 0) { - const env_var_t expansion = env_get(varname); + const env_var_t expansion = env_get_string(varname); if (!expansion.missing_or_empty()) { - abbreviations.emplace(std::make_pair(abbr, expansion.as_string())); + abbreviations.emplace(std::make_pair(abbr, expansion)); } } } diff --git a/src/function.cpp b/src/function.cpp index ac529d62d..c61f8bb47 100644 --- a/src/function.cpp +++ b/src/function.cpp @@ -78,7 +78,7 @@ static int load(const wcstring &name) { static void autoload_names(std::set &names, int get_hidden) { size_t i; - const env_var_t path_var_wstr = env_get(L"fish_function_path"); + const env_var_t path_var_wstr = env_get_string(L"fish_function_path"); if (path_var_wstr.missing()) return; const wchar_t *path_var = path_var_wstr.c_str(); @@ -121,7 +121,7 @@ void function_init() { static std::map snapshot_vars(const wcstring_list_t &vars) { std::map result; for (wcstring_list_t::const_iterator it = vars.begin(), end = vars.end(); it != end; ++it) { - result.insert(std::make_pair(*it, env_get(*it))); + result.insert(std::make_pair(*it, env_get_string(*it))); } return result; } diff --git a/src/highlight.cpp b/src/highlight.cpp index 69c9d0a45..8722bf7ad 100644 --- a/src/highlight.cpp +++ b/src/highlight.cpp @@ -221,12 +221,12 @@ static bool is_potential_cd_path(const wcstring &path, const wcstring &working_d directories.push_back(working_directory); } else { // Get the CDPATH. - env_var_t cdpath = env_get(L"CDPATH"); + env_var_t cdpath = env_get_string(L"CDPATH"); if (cdpath.missing_or_empty()) cdpath = L"."; // Tokenize it into directories. std::vector pathsv; - cdpath.to_list(pathsv); + tokenize_variable_array(cdpath, pathsv); for (auto next_path : pathsv) { if (next_path.empty()) next_path = L"."; // Ensure that we use the working directory for relative cdpaths like ".". @@ -268,17 +268,17 @@ rgb_color_t highlight_get_color(highlight_spec_t highlight, bool is_background) return rgb_color_t::normal(); } - env_var_t var = env_get(highlight_var[idx]); + env_var_t val_wstr = env_get_string(highlight_var[idx]); // debug( 1, L"%d -> %d -> %ls", highlight, idx, val ); - if (var.missing()) var = env_get(highlight_var[0]); + if (val_wstr.missing()) val_wstr = env_get_string(highlight_var[0]); - if (!var.missing()) result = parse_color(var.as_string(), treat_as_background); + if (!val_wstr.missing()) result = parse_color(val_wstr, treat_as_background); // Handle modifiers. if (highlight & highlight_modifier_valid_path) { - env_var_t val2_wstr = env_get(L"fish_color_valid_path"); + env_var_t val2_wstr = env_get_string(L"fish_color_valid_path"); const wcstring val2 = val2_wstr.missing() ? L"" : val2_wstr.c_str(); rgb_color_t result2 = parse_color(val2, is_background); diff --git a/src/history.cpp b/src/history.cpp index a293933b9..915e0b726 100644 --- a/src/history.cpp +++ b/src/history.cpp @@ -1796,9 +1796,8 @@ void history_sanity_check() { wcstring history_session_id() { wcstring result = DFLT_FISH_HISTORY_SESSION_ID; - const env_var_t var = env_get(L"FISH_HISTORY"); - if (!var.missing()) { - wcstring session_id = var.as_string(); + const env_var_t session_id = env_get_string(L"FISH_HISTORY"); + if (!session_id.missing()) { if (session_id.empty()) { result = L""; } else if (session_id == L"default") { diff --git a/src/input.cpp b/src/input.cpp index 94d4a4469..d4dc16698 100644 --- a/src/input.cpp +++ b/src/input.cpp @@ -201,8 +201,8 @@ static int input_function_args_index = 0; /// Return the current bind mode. wcstring input_get_bind_mode() { - env_var_t mode = env_get(FISH_BIND_MODE_VAR); - return mode.missing() ? DEFAULT_BIND_MODE : mode.as_string(); + env_var_t mode = env_get_string(FISH_BIND_MODE_VAR); + return mode.missing() ? DEFAULT_BIND_MODE : mode; } /// Set the current bind mode. diff --git a/src/input_common.cpp b/src/input_common.cpp index 22c402a7d..3ccb2324b 100644 --- a/src/input_common.cpp +++ b/src/input_common.cpp @@ -160,7 +160,7 @@ static wint_t readb() { // Update the wait_on_escape_ms value in response to the fish_escape_delay_ms user variable being // set. void update_wait_on_escape_ms() { - env_var_t escape_time_ms = env_get(L"fish_escape_delay_ms"); + env_var_t escape_time_ms = env_get_string(L"fish_escape_delay_ms"); if (escape_time_ms.missing_or_empty()) { wait_on_escape_ms = WAIT_ON_ESCAPE_DEFAULT; return; diff --git a/src/output.cpp b/src/output.cpp index 94252223b..41422b52b 100644 --- a/src/output.cpp +++ b/src/output.cpp @@ -554,7 +554,7 @@ void writembs_check(char *mbs, const char *mbs_name, const char *file, long line if (mbs != NULL) { tputs(mbs, 1, &writeb); } else { - env_var_t term = env_get(L"TERM"); + env_var_t term = env_get_string(L"TERM"); const wchar_t *fmt = _(L"Tried to use terminfo string %s on line %ld of %s, which is " L"undefined in terminal of type \"%ls\". Please report this error to %s"); diff --git a/src/path.cpp b/src/path.cpp index 1b20d8fc6..36997b76d 100644 --- a/src/path.cpp +++ b/src/path.cpp @@ -50,7 +50,7 @@ static bool path_get_path_core(const wcstring &cmd, wcstring *out_path, int err = ENOENT; wcstring bin_path; if (!bin_path_var.missing()) { - bin_path = bin_path_var.as_string(); + bin_path = bin_path_var; } else { // Note that PREFIX is defined in the `Makefile` and is thus defined when this module is // compiled. This ensures we always default to "/bin", "/usr/bin" and the bin dir defined @@ -104,7 +104,7 @@ bool path_get_path(const wcstring &cmd, wcstring *out_path, const env_vars_snaps } bool path_get_path(const wcstring &cmd, wcstring *out_path) { - return path_get_path_core(cmd, out_path, env_get(L"PATH")); + return path_get_path_core(cmd, out_path, env_get_string(L"PATH")); } wcstring_list_t path_get_paths(const wcstring &cmd) { @@ -122,9 +122,9 @@ wcstring_list_t path_get_paths(const wcstring &cmd) { return paths; } - env_var_t path_var = env_get(L"PATH"); + wcstring env_path = env_get_string(L"PATH"); std::vector pathsv; - path_var.to_list(pathsv); + tokenize_variable_array(env_path, pathsv); for (auto path : pathsv) { if (path.empty()) continue; append_path_component(path, cmd); @@ -141,11 +141,10 @@ wcstring_list_t path_get_paths(const wcstring &cmd) { return paths; } -bool path_get_cdpath(const env_var_t &dir_var, wcstring *out, const wchar_t *wd, +bool path_get_cdpath(const wcstring &dir, wcstring *out, const wchar_t *wd, const env_vars_snapshot_t &env_vars) { int err = ENOENT; - if (dir_var.missing_or_empty()) return false; - wcstring dir = dir_var.as_string(); + if (dir.empty()) return false; if (wd) { size_t len = wcslen(wd); @@ -169,7 +168,7 @@ bool path_get_cdpath(const env_var_t &dir_var, wcstring *out, const wchar_t *wd, if (cdpaths.missing_or_empty()) cdpaths = L"."; std::vector cdpathsv; - cdpaths.to_list(cdpathsv); + tokenize_variable_array(cdpaths, cdpathsv); for (auto next_path : cdpathsv) { if (next_path.empty()) next_path = L"."; if (next_path == L"." && wd != NULL) { @@ -288,20 +287,19 @@ static void path_create(wcstring &path, const wcstring &xdg_var, const wcstring // The vars we fetch must be exported. Allowing them to be universal doesn't make sense and // allowing that creates a lock inversion that deadlocks the shell since we're called before // uvars are available. - const env_var_t xdg_dir = env_get(xdg_var, ENV_GLOBAL | ENV_EXPORT); + const env_var_t xdg_dir = env_get_string(xdg_var, ENV_GLOBAL | ENV_EXPORT); if (!xdg_dir.missing_or_empty()) { using_xdg = true; - path = xdg_dir.as_string() + L"/fish"; + path = xdg_dir + L"/fish"; if (create_directory(path) != -1) { path_done = true; } else { saved_errno = errno; } } else { - const env_var_t home = env_get(L"HOME", ENV_GLOBAL | ENV_EXPORT); + const env_var_t home = env_get_string(L"HOME", ENV_GLOBAL | ENV_EXPORT); if (!home.missing_or_empty()) { - path = home.as_string() + - (which_dir == L"config" ? L"/.config/fish" : L"/.local/share/fish"); + path = home + (which_dir == L"config" ? L"/.config/fish" : L"/.local/share/fish"); if (create_directory(path) != -1) { path_done = true; } else { diff --git a/src/path.h b/src/path.h index 2b62dfa03..371d0ef86 100644 --- a/src/path.h +++ b/src/path.h @@ -61,7 +61,7 @@ wcstring_list_t path_get_paths(const wcstring &cmd); /// \param vars The environment variable snapshot to use (for the CDPATH variable) /// \return 0 if the command can not be found, the path of the command otherwise. The path should be /// free'd with free(). -bool path_get_cdpath(const env_var_t &dir, wcstring *out_or_NULL, const wchar_t *wd = NULL, +bool path_get_cdpath(const wcstring &dir, wcstring *out_or_NULL, const wchar_t *wd = NULL, const env_vars_snapshot_t &vars = env_vars_snapshot_t::current()); /// Returns whether the path can be used for an implicit cd command; if so, also returns the path by diff --git a/src/reader.cpp b/src/reader.cpp index 7daccce2a..0493dd2cb 100644 --- a/src/reader.cpp +++ b/src/reader.cpp @@ -2063,8 +2063,8 @@ void reader_import_history_if_necessary(void) { // Try opening a bash file. We make an effort to respect $HISTFILE; this isn't very complete // (AFAIK it doesn't have to be exported), and to really get this right we ought to ask bash // itself. But this is better than nothing. - const env_var_t var = env_get(L"HISTFILE"); - wcstring path = (var.missing() ? L"~/.bash_history" : var.as_string()); + const env_var_t var = env_get_string(L"HISTFILE"); + wcstring path = (var.missing() ? L"~/.bash_history" : var); expand_tilde(path); FILE *f = wfopen(path, "r"); if (f) { diff --git a/src/screen.cpp b/src/screen.cpp index 9b0988276..0d0d3cc1d 100644 --- a/src/screen.cpp +++ b/src/screen.cpp @@ -120,8 +120,8 @@ static bool is_screen_name_escape_seq(const wchar_t *code, size_t *resulting_len #if 0 // TODO: Decide if this should be removed or modified to also test for TERM values that begin // with "tmux". See issue #3512. - const env_var_t term_name = env_get(L"TERM"); - if (term_name.missing_or_empty() || !string_prefixes_string(L"screen", term_name)) { + const env_var_t term_name = env_get_string(L"TERM"); + if (term_name.missing() || !string_prefixes_string(L"screen", term_name)) { return false; } #endif From 088b21a15feac36955bd3c7b8d62b7a9bb55be14 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:32:16 -0700 Subject: [PATCH 74/79] Revert "use the new `set -a` and `set -p` in our scripts" This reverts commit 8b79f4e5c93acf7e643ef5b7b854fa7ad5a39487. It was meant for the major branch. --- share/functions/__fish_bind_test1.fish | 3 ++- share/functions/__fish_bind_test2.fish | 3 ++- share/functions/__fish_complete_mount_opts.fish | 8 ++++---- share/functions/__fish_complete_proc.fish | 16 ++++++++-------- .../__fish_complete_subcommand_root.fish | 4 +++- share/functions/__fish_contains_opt.fish | 4 ++-- share/functions/__fish_git_prompt.fish | 8 ++++---- share/functions/__fish_hg_prompt.fish | 12 ++++++------ .../__fish_make_completion_signals.fish | 7 +++---- share/functions/__fish_print_filesystems.fish | 5 +++-- share/functions/__fish_print_hostnames.fish | 4 ++-- share/functions/__fish_set_locale.fish | 6 +++--- share/functions/__terlar_git_prompt.fish | 14 +++++++------- share/functions/abbr.fish | 4 ++-- share/functions/cd.fish | 2 +- share/functions/cdh.fish | 2 +- share/functions/help.fish | 6 +++--- share/functions/history.fish | 8 ++++---- share/functions/ls.fish | 2 +- share/functions/pushd.fish | 2 +- share/functions/realpath.fish | 4 ++-- share/functions/string.fish | 2 +- share/functions/umask.fish | 4 ++-- 23 files changed, 67 insertions(+), 63 deletions(-) diff --git a/share/functions/__fish_bind_test1.fish b/share/functions/__fish_bind_test1.fish index a636ecd89..fa5fe4de1 100644 --- a/share/functions/__fish_bind_test1.fish +++ b/share/functions/__fish_bind_test1.fish @@ -1,3 +1,4 @@ + function __fish_bind_test1 set -l args set -l use_keys no @@ -9,7 +10,7 @@ function __fish_bind_test1 case "-*" case "*" - set -a args $i + set args $args $i end end diff --git a/share/functions/__fish_bind_test2.fish b/share/functions/__fish_bind_test2.fish index 74f594fcb..fded8b262 100644 --- a/share/functions/__fish_bind_test2.fish +++ b/share/functions/__fish_bind_test2.fish @@ -1,3 +1,4 @@ + function __fish_bind_test2 set -l args for i in (commandline -poc) @@ -5,7 +6,7 @@ function __fish_bind_test2 case "-*" case "*" - set -a args $i + set args $args $i end end diff --git a/share/functions/__fish_complete_mount_opts.fish b/share/functions/__fish_complete_mount_opts.fish index 9afc4020b..2a98c18a2 100644 --- a/share/functions/__fish_complete_mount_opts.fish +++ b/share/functions/__fish_complete_mount_opts.fish @@ -145,13 +145,13 @@ function __fish_complete_mount_opts switch (string replace -r '=.*' '=' -- $last_arg) case uid= - set -a fish_mount_opts uid=(__fish_print_user_ids) + set fish_mount_opts $fish_mount_opts uid=(__fish_print_user_ids) case gid= - set -a fish_mount_opts gid=(__fish_print_group_ids) + set fish_mount_opts $fish_mount_opts gid=(__fish_print_group_ids) case setuid= - set -a fish_mount_opts setuid=(__fish_print_user_ids) + set fish_mount_opts $fish_mount_opts setuid=(__fish_print_user_ids) case setgid= - set -a fish_mount_opts setgid=(__fish_print_group_ids) + set fish_mount_opts $fish_mount_opts setgid=(__fish_print_group_ids) end diff --git a/share/functions/__fish_complete_proc.fish b/share/functions/__fish_complete_proc.fish index 8fb5d7260..ab92bdfb3 100644 --- a/share/functions/__fish_complete_proc.fish +++ b/share/functions/__fish_complete_proc.fish @@ -14,33 +14,33 @@ function __fish_complete_proc --description 'Complete by list of running process set ps_opt -A -o command # Erase everything after the first space - set -a sed_cmds 's/ .*//' + set sed_cmds $sed_cmds 's/ .*//' # Erases weird stuff Linux gives like kworker/0:0 - set -a sed_cmds 's|/[0-9]:[0-9]]$||g' + set sed_cmds $sed_cmds 's|/[0-9]:[0-9]]$||g' # Retain the last path component only - set -a sed_cmds 's|.*/||g' + set sed_cmds $sed_cmds 's|.*/||g' # Strip off square brackets. Cute, huh? - set -a sed_cmds 's/[][]//g' + set sed_cmds $sed_cmds 's/[][]//g' # Erase things that are just numbers - set -a sed_cmds 's/^[0-9]*$//' + set sed_cmds $sed_cmds 's/^[0-9]*$//' else # OS X, BSD. Preserve leading spaces. set ps_opt axc -o comm # Delete parenthesized (zombie) processes - set -a sed_cmds '/(.*)/d' + set sed_cmds $sed_cmds '/(.*)/d' end # Append sed command to delete first line (the header) - set -a sed_cmds '1d' + set sed_cmds $sed_cmds '1d' # Append sed commands to delete leading dashes and trailing spaces # In principle, commands may have trailing spaces, but ps emits space padding on OS X - set -a sed_cmds 's/^-//' 's/ *$//' + set sed_cmds $sed_cmds 's/^-//' 's/ *$//' # Run ps, pipe it through our massive set of sed commands, then sort and unique ps $ps_opt | sed '-e '$sed_cmds | sort -u diff --git a/share/functions/__fish_complete_subcommand_root.fish b/share/functions/__fish_complete_subcommand_root.fish index 736abeeff..f27984543 100644 --- a/share/functions/__fish_complete_subcommand_root.fish +++ b/share/functions/__fish_complete_subcommand_root.fish @@ -1,4 +1,6 @@ + + function __fish_complete_subcommand_root -d "Run the __fish_complete_subcommand function using a PATH containing /sbin and /usr/sbin" - set -lx -p PATH /sbin /usr/sbin ^/dev/null + set -lx PATH /sbin /usr/sbin $PATH ^/dev/null __fish_complete_subcommand $argv end diff --git a/share/functions/__fish_contains_opt.fish b/share/functions/__fish_contains_opt.fish index 0d10d6d23..ffd139ad6 100644 --- a/share/functions/__fish_contains_opt.fish +++ b/share/functions/__fish_contains_opt.fish @@ -6,7 +6,7 @@ function __fish_contains_opt -d "Checks if a specific option has been given in t for i in $argv if test -n "$next_short" set next_short - set -a short_opt $i + set short_opt $short_opt $i else switch $i case -s @@ -15,7 +15,7 @@ function __fish_contains_opt -d "Checks if a specific option has been given in t echo __fish_contains_opt: Unknown option $i >&2 return 1 case '*' - set -a long_opt $i + set long_opt $long_opt $i end end end diff --git a/share/functions/__fish_git_prompt.fish b/share/functions/__fish_git_prompt.fish index 8416806f8..1b4b797f5 100644 --- a/share/functions/__fish_git_prompt.fish +++ b/share/functions/__fish_git_prompt.fish @@ -724,7 +724,7 @@ end set -l varargs for var in repaint describe_style show_informative_status showdirtystate showstashstate showuntrackedfiles showupstream - set -a varargs --on-variable __fish_git_prompt_$var + set varargs $varargs --on-variable __fish_git_prompt_$var end function __fish_git_prompt_repaint $varargs --description "Event handler, repaints prompt when functionality changes" if status --is-interactive @@ -741,9 +741,9 @@ end set -l varargs for var in '' _prefix _suffix _bare _merging _cleanstate _invalidstate _upstream _flags _branch _dirtystate _stagedstate _branch_detached _stashstate _untrackedfiles - set -a varargs --on-variable __fish_git_prompt_color$var + set varargs $varargs --on-variable __fish_git_prompt_color$var end -set -a varargs --on-variable __fish_git_prompt_showcolorhints +set varargs $varargs --on-variable __fish_git_prompt_showcolorhints function __fish_git_prompt_repaint_color $varargs --description "Event handler, repaints prompt when any color changes" if status --is-interactive set -l var $argv[3] @@ -762,7 +762,7 @@ end set -l varargs for var in cleanstate dirtystate invalidstate stagedstate stashstate stateseparator untrackedfiles upstream_ahead upstream_behind upstream_diverged upstream_equal upstream_prefix - set -a varargs --on-variable __fish_git_prompt_char_$var + set varargs $varargs --on-variable __fish_git_prompt_char_$var end function __fish_git_prompt_repaint_char $varargs --description "Event handler, repaints prompt when any char changes" if status --is-interactive diff --git a/share/functions/__fish_hg_prompt.fish b/share/functions/__fish_hg_prompt.fish index 0ab444514..ee02d2b9b 100644 --- a/share/functions/__fish_hg_prompt.fish +++ b/share/functions/__fish_hg_prompt.fish @@ -70,17 +70,17 @@ function __fish_hg_prompt --description 'Write out the hg prompt' # Add a character for each file status if we have one switch $line case 'A ' - set -a hg_statuses added + set hg_statuses $hg_statuses added case 'M ' ' M' - set -a hg_statuses modified + set hg_statuses $hg_statuses modified case 'C ' - set -a hg_statuses copied + set hg_statuses $hg_statuses copied case 'D ' ' D' - set -a hg_statuses deleted + set hg_statuses $hg_statuses deleted case '\? ' - set -a hg_statuses untracked + set hg_statuses $hg_statuses untracked case 'U*' '*U' 'DD' 'AA' - set -a hg_statuses unmerged + set hg_statuses $hg_statuses unmerged end end diff --git a/share/functions/__fish_make_completion_signals.fish b/share/functions/__fish_make_completion_signals.fish index cac6e135f..2dd88ac47 100644 --- a/share/functions/__fish_make_completion_signals.fish +++ b/share/functions/__fish_make_completion_signals.fish @@ -2,8 +2,6 @@ function __fish_make_completion_signals --description 'Make list of kill signals set -q __kill_signals and return 0 - set -g __kill_signals - # Some systems use the GNU coreutils kill command where `kill -L` produces an extended table # format that looks like this: # @@ -18,17 +16,18 @@ function __fish_make_completion_signals --description 'Make list of kill signals # Posix systems print out the name of a signal using 'kill -l SIGNUM'. complete -c kill -s l --description "List names of available signals" for i in (seq 31) - set -a __kill_signals $i" "(kill -l $i | tr '[:lower:]' '[:upper:]') + set -g __kill_signals $__kill_signals $i" "(kill -l $i | tr '[:lower:]' '[:upper:]') end else # Debian and some related systems use 'kill -L' to write out a numbered list # of signals. Use this to complete on both number _and_ on signal name. complete -c kill -s L --description "List codes and names of available signals" + set -g __kill_signals kill -L | sed -e 's/^ //; s/ */ /g; y/ /\n/' | while read -l signo test -z "$signo" and break # the sed above produces one blank line at the end read -l signame - set -a __kill_signals "$signo $signame" + set -g __kill_signals $__kill_signals "$signo $signame" end end end diff --git a/share/functions/__fish_print_filesystems.fish b/share/functions/__fish_print_filesystems.fish index d329ecd0c..b14be36bc 100644 --- a/share/functions/__fish_print_filesystems.fish +++ b/share/functions/__fish_print_filesystems.fish @@ -1,7 +1,8 @@ + function __fish_print_filesystems -d "Print a list of all known filesystem types" set -l fs adfs affs autofs coda coherent cramfs devpts efs ext ext2 ext3 - set -a fs hfs hpfs iso9660 jfs minix msdos ncpfs nfs ntfs proc qnx4 ramfs - set -a fs reiserfs romfs smbfs sysv tmpfs udf ufs umsdos vfat xenix xfs xiafs + set fs $fs hfs hpfs iso9660 jfs minix msdos ncpfs nfs ntfs proc qnx4 ramfs + set fs $fs reiserfs romfs smbfs sysv tmpfs udf ufs umsdos vfat xenix xfs xiafs # Mount has helper binaries to mount filesystems # These are called mount.* and are placed somewhere in $PATH set -l mountfs $PATH/mount.* $PATH/mount_* diff --git a/share/functions/__fish_print_hostnames.fish b/share/functions/__fish_print_hostnames.fish index ebe160a74..efa3cac4f 100644 --- a/share/functions/__fish_print_hostnames.fish +++ b/share/functions/__fish_print_hostnames.fish @@ -53,7 +53,7 @@ function __fish_print_hostnames -d "Print a list of known hostnames" set -l orig_dir $PWD set -l paths for config in $argv - set -a paths (cat $config ^/dev/null \ + set paths $paths (cat $config ^/dev/null \ # Keep only Include lines | string match -r -i '^\s*Include\s+.+' \ # Remove Include syntax @@ -91,7 +91,7 @@ function __fish_print_hostnames -d "Print a list of known hostnames" # Print hosts from system wide ssh configuration file string match -r -i '^\s*Host\s+\S+' <$file | string replace -r -i '^\s*Host\s+' '' | string trim | string replace -r '\s+' ' ' | string split ' ' | string match -v '*\**' # Extract known_host paths. - set -a known_hosts (string match -ri '^\s*UserKnownHostsFile|^\s*GlobalKnownHostsFile' <$file | string replace -ri '.*KnownHostsFile\s*' '') + set known_hosts $known_hosts (string match -ri '^\s*UserKnownHostsFile|^\s*GlobalKnownHostsFile' <$file | string replace -ri '.*KnownHostsFile\s*' '') end end for file in $known_hosts diff --git a/share/functions/__fish_set_locale.fish b/share/functions/__fish_set_locale.fish index cb0e28674..125b79d08 100644 --- a/share/functions/__fish_set_locale.fish +++ b/share/functions/__fish_set_locale.fish @@ -10,9 +10,9 @@ function __fish_set_locale set -l LOCALE_VARS - set -a LOCALE_VARS LANG LANGUAGE LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE - set -a LOCALE_VARS LC_MONETARY LC_MESSAGES LC_PAPER LC_NAME LC_ADDRESS - set -a LOCALE_VARS LC_TELEPHONE LC_MEASUREMENT LC_IDENTIFICATION + set LOCALE_VARS $LOCALE_VARS LANG LANGUAGE LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE + set LOCALE_VARS $LOCALE_VARS LC_MONETARY LC_MESSAGES LC_PAPER LC_NAME LC_ADDRESS + set LOCALE_VARS $LOCALE_VARS LC_TELEPHONE LC_MEASUREMENT LC_IDENTIFICATION # We check LC_ALL to figure out if we have a locale but we don't set it later. That is because # locale.conf doesn't allow it so we should not set it. diff --git a/share/functions/__terlar_git_prompt.fish b/share/functions/__terlar_git_prompt.fish index f1fcedf33..46013d928 100644 --- a/share/functions/__terlar_git_prompt.fish +++ b/share/functions/__terlar_git_prompt.fish @@ -52,19 +52,19 @@ function __terlar_git_prompt --description 'Write out the git prompt' switch $i case 'A ' - set -a gs added + set gs $gs added case 'M ' ' M' - set -a gs modified + set gs $gs modified case 'R ' - set -a gs renamed + set gs $gs renamed case 'C ' - set -a gs copied + set gs $gs copied case 'D ' ' D' - set -a gs deleted + set gs $gs deleted case '\?\?' - set -a gs untracked + set gs $gs untracked case 'U*' '*U' 'DD' 'AA' - set -a gs unmerged + set gs $gs unmerged end end diff --git a/share/functions/abbr.fish b/share/functions/abbr.fish index 33e249502..508965fd6 100644 --- a/share/functions/abbr.fish +++ b/share/functions/abbr.fish @@ -1,7 +1,7 @@ function abbr --description "Manage abbreviations using new fish 3.0 scheme." set -l options --stop-nonopt --exclusive 'a,r,e,l,s' --exclusive 'g,U' - set -a options 'h/help' 'a/add' 'r/rename' 'e/erase' 'l/list' 's/show' - set -a options 'g/global' 'U/universal' + set options $options 'h/help' 'a/add' 'r/rename' 'e/erase' 'l/list' 's/show' + set options $options 'g/global' 'U/universal' argparse -n abbr $options -- $argv or return diff --git a/share/functions/cd.fish b/share/functions/cd.fish index 23f169d2c..9f426c031 100644 --- a/share/functions/cd.fish +++ b/share/functions/cd.fish @@ -35,7 +35,7 @@ function cd --description "Change directory" or set -l dirprev set -q dirprev[$MAX_DIR_HIST] and set -e dirprev[1] - set -g -a dirprev $previous + set -g dirprev $dirprev $previous set -e dirnext set -g __fish_cd_direction prev end diff --git a/share/functions/cdh.fish b/share/functions/cdh.fish index c6a56176e..773db10ba 100644 --- a/share/functions/cdh.fish +++ b/share/functions/cdh.fish @@ -26,7 +26,7 @@ function cdh --description "Menu based cd command" set -l uniq_dirs for dir in $all_dirs[-1..1] if not contains $dir $uniq_dirs - set -a uniq_dirs $dir + set uniq_dirs $uniq_dirs $dir end end diff --git a/share/functions/help.fish b/share/functions/help.fish index 30cf5d9d9..d8c4f26c6 100644 --- a/share/functions/help.fish +++ b/share/functions/help.fish @@ -10,9 +10,9 @@ function help --description 'Show help for the fish shell' set -l fish_help_item $argv[1] set -l help_topics syntax completion editor job-control todo bugs history killring help - set -a help_topics color prompt title variables builtin-overview changes expand - set -a help_topics expand-variable expand-home expand-brace expand-wildcard - set -a help_topics expand-command-substitution expand-process + set help_topics $help_topics color prompt title variables builtin-overview changes expand + set help_topics $help_topics expand-variable expand-home expand-brace expand-wildcard + set help_topics $help_topics expand-command-substitution expand-process # # Find a suitable browser for viewing the help pages. This is needed diff --git a/share/functions/history.fish b/share/functions/history.fish index 7e90caa99..bfe2651ad 100644 --- a/share/functions/history.fish +++ b/share/functions/history.fish @@ -19,14 +19,14 @@ function history --description "display or manipulate interactive command histor set -l cmd history set -l options --exclusive 'c,e,p' --exclusive 'S,D,M,V,C' --exclusive 't,T' - set -a options 'h/help' 'c/contains' 'e/exact' 'p/prefix' - set -a options 'C/case-sensitive' 'z/null' 't/show-time=?' 'n#max' + set options $options 'h/help' 'c/contains' 'e/exact' 'p/prefix' + set options $options 'C/case-sensitive' 'z/null' 't/show-time=?' 'n#max' # This long option is deprecated and here solely for legacy compatibility. People should use # -t or --show-time now. - set -a options 'T-with-time=?' + set options $options 'T-with-time=?' # The following options are deprecated and will be removed in the next major release. # Note that they do not have usable short flags. - set -a options 'S-search' 'D-delete' 'M-merge' 'V-save' 'R-clear' + set options $options 'S-search' 'D-delete' 'M-merge' 'V-save' 'R-clear' argparse -n $cmd $options -- $argv or return diff --git a/share/functions/ls.fish b/share/functions/ls.fish index a51b99c8d..4a6b0db3d 100644 --- a/share/functions/ls.fish +++ b/share/functions/ls.fish @@ -6,7 +6,7 @@ if command ls --version >/dev/null ^/dev/null function ls --description "List contents of directory" set -l param --color=auto if isatty 1 - set -a param --indicator-style=classify + set param $param --indicator-style=classify end command ls $param $argv end diff --git a/share/functions/pushd.fish b/share/functions/pushd.fish index 52af5c466..0ccf01af9 100644 --- a/share/functions/pushd.fish +++ b/share/functions/pushd.fish @@ -77,6 +77,6 @@ function pushd --description 'Push directory to stack' end # argv[1] is a directory - set -g -p dirstack $PWD + set -g dirstack $PWD $dirstack cd $argv[1] end diff --git a/share/functions/realpath.fish b/share/functions/realpath.fish index 4ae3b15c8..3dc17451f 100644 --- a/share/functions/realpath.fish +++ b/share/functions/realpath.fish @@ -23,8 +23,8 @@ end function realpath -d "return an absolute path without symlinks" set -l options 'h/help' 'q/quiet' 'V-version' 's/strip' 'N-no-symlinks' 'z/zero' - set -a options 'e/canonicalize-existing' 'm/canonicalize-missing' 'L/logical' 'P/physical' - set -a options 'R-relative-to=' 'B-relative-base=' + set options $options 'e/canonicalize-existing' 'm/canonicalize-missing' 'L/logical' 'P/physical' + set options $options 'R-relative-to=' 'B-relative-base=' argparse -n realpath --min-args=1 $options -- $argv or return diff --git a/share/functions/string.fish b/share/functions/string.fish index 72cb67a62..799258217 100644 --- a/share/functions/string.fish +++ b/share/functions/string.fish @@ -14,7 +14,7 @@ if not contains string (builtin -n) set -g __is_launched_without_string 1 end end - set -p PATH $__fish_bin_dir + set PATH $__fish_bin_dir $PATH set string_cmd string \'$argv\' if fish -c 'contains string (builtin -n)' diff --git a/share/functions/umask.fish b/share/functions/umask.fish index d6147d680..1ae450372 100644 --- a/share/functions/umask.fish +++ b/share/functions/umask.fish @@ -87,9 +87,9 @@ function __fish_umask_parse string match -q '*u*' $scope and set scopes_to_modify 1 string match -q '*g*' $scope - and set -a scopes_to_modify 2 + and set scopes_to_modify $scopes_to_modify 2 string match -q '*o*' $scope - and set -a scopes_to_modify 3 + and set scopes_to_modify $scopes_to_modify 3 string match -q '*a*' $scope and set scopes_to_modify 1 2 3 From 9cb25544230516078f07fae81cd787da90a71351 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:33:50 -0700 Subject: [PATCH 75/79] Revert "implement `set --append` and `set --prepend`" This reverts commit 67de733b9b2262d424fa1f656157c952a9168b2b. It was meant for the major branch. --- doc_src/set.txt | 23 ++----- src/builtin_set.cpp | 147 +++++++++++++++----------------------------- tests/set.err | 9 --- tests/set.in | 31 ---------- tests/set.out | 59 ------------------ tests/test8.err | 21 ------- tests/test8.in | 14 ++--- tests/test8.out | 26 ++------ 8 files changed, 68 insertions(+), 262 deletions(-) diff --git a/doc_src/set.txt b/doc_src/set.txt index e335abd99..c68963aac 100644 --- a/doc_src/set.txt +++ b/doc_src/set.txt @@ -21,10 +21,6 @@ With both variable names and values provided, `set` assigns the variable `VARIAB The following options control variable scope: -- `-a` or `--append` causes the values to be appended to the current set of values for the variable. This can be used with `--prepend` to both append and prepend at the same time. This cannot be used when assigning to a variable slice. - -- `-p` or `--prepend` causes the values to be prepended to the current set of values for the variable. This can be used with `--append` to both append and prepend at the same time. This cannot be used when assigning to a variable slice. - - `-l` or `--local` forces the specified shell variable to be given a scope that is local to the current block, even if a variable with the given name exists and is non-local - `-g` or `--global` causes the specified shell variable to be given a global scope. Non-global variables disappear when the block they belong to ends @@ -80,31 +76,24 @@ In erase mode, if variable indices are specified, only the specified slices of t In assignment mode, `set` does not modify the exit status. This allows simultaneous capture of the output and exit status of a subcommand, e.g. `if set output (command)`. In query mode, the exit status is the number of variables that were not found. In erase mode, `set` exits with a zero exit status in case of success, with a non-zero exit status if the commandline was invalid, if the variable was write-protected or if the variable did not exist. -\subsection set-examples Examples +\subsection set-example Example \fish -# Prints all global, exported variables. set -xg +# Prints all global, exported variables. +set foo hi # Sets the value of the variable $foo to be 'hi'. -set foo hi -# Appends the value "there" to the variable $foo. -set -a foo there - -# Does the same thing as the previous two commands the way it would be done pre-fish 3.0. -set foo hi -set foo $foo there - -# Removes the variable $smurf set -e smurf +# Removes the variable $smurf -# Changes the fourth element of the $PATH array to ~/bin set PATH[4] ~/bin +# Changes the fourth element of the $PATH array to ~/bin -# Outputs the path to Python if `type -p` returns true. if set python_path (type -p python) echo "Python is at $python_path" end +# Outputs the path to Python if `type -p` returns true. \endfish \subsection set-notes Notes diff --git a/src/builtin_set.cpp b/src/builtin_set.cpp index daa713a99..a5f3aada7 100644 --- a/src/builtin_set.cpp +++ b/src/builtin_set.cpp @@ -40,22 +40,19 @@ struct set_cmd_opts_t { bool universal = false; bool query = false; bool shorten_ok = true; - bool append = false; - bool prepend = false; bool preserve_failure_exit_status = true; }; // Variables used for parsing the argument list. This command is atypical in using the "+" // (REQUIRE_ORDER) option for flag parsing. This is not typical of most fish commands. It means // we stop scanning for flags when the first non-flag argument is seen. -static const wchar_t *short_options = L"+:LSUaeghlnpqux"; +static const wchar_t *short_options = L"+:LSUeghlnqux"; static const struct woption long_options[] = { {L"export", no_argument, NULL, 'x'}, {L"global", no_argument, NULL, 'g'}, {L"local", no_argument, NULL, 'l'}, {L"erase", no_argument, NULL, 'e'}, {L"names", no_argument, NULL, 'n'}, {L"unexport", no_argument, NULL, 'u'}, {L"universal", no_argument, NULL, 'U'}, {L"long", no_argument, NULL, 'L'}, {L"query", no_argument, NULL, 'q'}, {L"show", no_argument, NULL, 'S'}, - {L"append", no_argument, NULL, 'a'}, {L"prepend", no_argument, NULL, 'p'}, {L"help", no_argument, NULL, 'h'}, {NULL, 0, NULL, 0}}; // Error message for invalid path operations. @@ -80,45 +77,28 @@ static int parse_cmd_opts(set_cmd_opts_t &opts, int *optind, //!OCLINT(high ncs wgetopter_t w; while ((opt = w.wgetopt_long(argc, argv, short_options, long_options, NULL)) != -1) { switch (opt) { - case 'a': { - opts.append = true; - break; - } case 'e': { opts.erase = true; opts.preserve_failure_exit_status = false; break; } - case 'g': { - opts.global = true; - break; - } - case 'h': { - opts.print_help = true; - break; - } - case 'l': { - opts.local = true; - break; - } case 'n': { opts.list = true; opts.preserve_failure_exit_status = false; break; } - case 'p': { - opts.prepend = true; - break; - } - case 'q': { - opts.query = true; - opts.preserve_failure_exit_status = false; - break; - } case 'x': { opts.exportv = true; break; } + case 'l': { + opts.local = true; + break; + } + case 'g': { + opts.global = true; + break; + } case 'u': { opts.unexport = true; break; @@ -131,11 +111,20 @@ static int parse_cmd_opts(set_cmd_opts_t &opts, int *optind, //!OCLINT(high ncs opts.shorten_ok = false; break; } + case 'q': { + opts.query = true; + opts.preserve_failure_exit_status = false; + break; + } case 'S': { opts.show = true; opts.preserve_failure_exit_status = false; break; } + case 'h': { + opts.print_help = true; + break; + } case ':': { builtin_missing_argument(parser, streams, cmd, argv[w.woptind - 1]); return STATUS_INVALID_ARGS; @@ -440,8 +429,8 @@ static int compute_scope(set_cmd_opts_t &opts) { /// Print the names of all environment variables in the scope. It will include the values unless the /// `set --list` flag was used. -static int builtin_set_list(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, wchar_t **argv, - parser_t &parser, io_streams_t &streams) { +static int builtin_set_list(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, + wchar_t **argv, parser_t &parser, io_streams_t &streams) { UNUSED(cmd); UNUSED(argc); UNUSED(argv); @@ -480,8 +469,8 @@ static int builtin_set_list(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, } // Query mode. Return the number of variables that do not exist out of the specified variables. -static int builtin_set_query(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, wchar_t **argv, - parser_t &parser, io_streams_t &streams) { +static int builtin_set_query(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, + wchar_t **argv, parser_t &parser, io_streams_t &streams) { int retval = 0; int scope = compute_scope(opts); @@ -604,8 +593,8 @@ static int builtin_set_show(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, } /// Erase a variable. -static int builtin_set_erase(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, wchar_t **argv, - parser_t &parser, io_streams_t &streams) { +static int builtin_set_erase(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, + wchar_t **argv, parser_t &parser, io_streams_t &streams) { if (argc != 1) { streams.err.append_format(BUILTIN_ERR_ARG_COUNT2, cmd, L"--erase", 1, argc); builtin_print_help(parser, streams, cmd, streams.err); @@ -644,111 +633,75 @@ static int builtin_set_erase(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, return check_global_scope_exists(cmd, opts, dest, streams); } -/// This handles the common case of setting the entire var to a set of values. -static int set_var_array(const wchar_t *cmd, set_cmd_opts_t &opts, const wchar_t *varname, - wcstring_list_t &new_values, int argc, wchar_t **argv, parser_t &parser, - io_streams_t &streams) { - UNUSED(cmd); - UNUSED(parser); - UNUSED(streams); - - if (opts.prepend || opts.append) { - int scope = compute_scope(opts); - if (opts.prepend) { - for (int i = 0; i < argc; i++) new_values.push_back(argv[i]); - } - - env_var_t var_str = env_get_string(varname, scope); - wcstring_list_t var_array; - if (!var_str.missing()) tokenize_variable_array(var_str, var_array); - new_values.insert(new_values.end(), var_array.begin(), var_array.end()); - - if (opts.append) { - for (int i = 0; i < argc; i++) new_values.push_back(argv[i]); - } - } else { - for (int i = 0; i < argc; i++) new_values.push_back(argv[i]); - } - - return STATUS_CMD_OK; -} - /// This handles the more difficult case of setting individual slices of a var. -static int set_var_slices(const wchar_t *cmd, set_cmd_opts_t &opts, const wchar_t *varname, - wcstring_list_t &new_values, std::vector &indexes, int argc, - wchar_t **argv, parser_t &parser, io_streams_t &streams) { +static int set_slices(const wchar_t *cmd, set_cmd_opts_t &opts, const wchar_t *dest, + std::vector &indexes, int argc, wchar_t **argv, parser_t &parser, + io_streams_t &streams) { UNUSED(parser); - if (opts.append || opts.prepend) { - streams.err.append_format( - L"%ls: Cannot use --append or --prepend when assigning to a slice", cmd); - builtin_print_help(parser, streams, cmd, streams.err); - return STATUS_INVALID_ARGS; - } - if (indexes.size() != static_cast(argc)) { streams.err.append_format(BUILTIN_SET_MISMATCHED_ARGS, cmd, indexes.size(), argc); } int scope = compute_scope(opts); // calculate the variable scope based on the provided options - const env_var_t var_str = env_get_string(varname, scope); - if (!var_str.missing()) tokenize_variable_array(var_str, new_values); + wcstring_list_t result; + const env_var_t dest_str = env_get_string(dest, scope); + if (!dest_str.missing()) tokenize_variable_array(dest_str, result); // Slice indexes have been calculated, do the actual work. - wcstring_list_t result; - for (int i = 0; i < argc; i++) result.push_back(argv[i]); + wcstring_list_t new_values; + for (int i = 0; i < argc; i++) new_values.push_back(argv[i]); - int retval = update_values(new_values, indexes, result); + int retval = update_values(result, indexes, new_values); if (retval != STATUS_CMD_OK) { streams.err.append_format(BUILTIN_SET_ARRAY_BOUNDS_ERR, cmd); return retval; } - return STATUS_CMD_OK; + retval = my_env_set(cmd, dest, result, scope, streams); + if (retval != STATUS_CMD_OK) return retval; + return check_global_scope_exists(cmd, opts, dest, streams); } /// Set a variable. -static int builtin_set_set(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, wchar_t **argv, - parser_t &parser, io_streams_t &streams) { +static int builtin_set_set(const wchar_t *cmd, set_cmd_opts_t &opts, int argc, + wchar_t **argv, parser_t &parser, io_streams_t &streams) { if (argc == 0) { streams.err.append_format(BUILTIN_ERR_MIN_ARG_COUNT1, cmd, 1); builtin_print_help(parser, streams, cmd, streams.err); return STATUS_INVALID_ARGS; } + int retval; int scope = compute_scope(opts); // calculate the variable scope based on the provided options - wchar_t *varname = argv[0]; + wchar_t *dest = argv[0]; argv++; argc--; std::vector indexes; - int idx_count = parse_index(indexes, varname, scope, streams); + int idx_count = parse_index(indexes, dest, scope, streams); if (idx_count == -1) { builtin_print_help(parser, streams, cmd, streams.err); return STATUS_INVALID_ARGS; } - if (!valid_var_name(varname)) { - streams.err.append_format(BUILTIN_ERR_VARNAME, cmd, varname); + if (!valid_var_name(dest)) { + streams.err.append_format(BUILTIN_ERR_VARNAME, cmd, dest); builtin_print_help(parser, streams, cmd, streams.err); return STATUS_INVALID_ARGS; } - int retval; - wcstring_list_t new_values; - if (idx_count == 0) { - // Handle the simple, common, case. Set the var to the specified values. - retval = set_var_array(cmd, opts, varname, new_values, argc, argv, parser, streams); - } else { + if (idx_count != 0) { // Handle the uncommon case of setting specific slices of a var. - retval = - set_var_slices(cmd, opts, varname, new_values, indexes, argc, argv, parser, streams); + return set_slices(cmd, opts, dest, indexes, argc, argv, parser, streams); } - if (retval != STATUS_CMD_OK) return retval; - retval = my_env_set(cmd, varname, new_values, scope, streams); + // This is the simple, common, case. Set the var to the specified values. + wcstring_list_t values; + for (int i = 0; i < argc; i++) values.push_back(argv[i]); + retval = my_env_set(cmd, dest, values, scope, streams); if (retval != STATUS_CMD_OK) return retval; - return check_global_scope_exists(cmd, opts, varname, streams); + return check_global_scope_exists(cmd, opts, dest, streams); } /// The set builtin creates, updates, and erases (removes, deletes) variables. diff --git a/tests/set.err b/tests/set.err index d6e567403..a6bbd2aee 100644 --- a/tests/set.err +++ b/tests/set.err @@ -5,12 +5,3 @@ $argle bargle: invalid var name #################### # Verify behavior of `set --show` - -#################### -# Appending works - -#################### -# Prepending works - -#################### -# Appending and prepending at same time works diff --git a/tests/set.in b/tests/set.in index 74ddb9232..b4fed0d36 100644 --- a/tests/set.in +++ b/tests/set.in @@ -13,34 +13,3 @@ set --show var1 set -g var2 set --show _unset_var var2 - -logmsg Appending works -set -g var3a a b c -set -a var3a -set -a var3a d -set -a var3a e f -set --show var3a -set -g var3b -set -a var3b -set --show var3b -set -g var3c -set -a var3c 'one string' -set --show var3c - -logmsg Prepending works -set -g var4a a b c -set -p var4a -set -p var4a d -set -p var4a e f -set --show var4a -set -g var4b -set -p var4b -set --show var4b -set -g var4c -set -p var4c 'one string' -set --show var4c - -logmsg Appending and prepending at same time works -set -g var5 abc def -set -a -p var5 0 x 0 -set --show var5 diff --git a/tests/set.out b/tests/set.out index b094b0107..41108e1d5 100644 --- a/tests/set.out +++ b/tests/set.out @@ -24,62 +24,3 @@ $var2: not set in local scope $var2: set in global scope, unexported, with 0 elements $var2: not set in universal scope - -#################### -# Appending works -$var3a: not set in local scope -$var3a: set in global scope, unexported, with 6 elements -$var3a[1]: length=1 value=|a| -$var3a[2]: length=1 value=|b| -$var3a[3]: length=1 value=|c| -$var3a[4]: length=1 value=|d| -$var3a[5]: length=1 value=|e| -$var3a[6]: length=1 value=|f| -$var3a: not set in universal scope - -$var3b: not set in local scope -$var3b: set in global scope, unexported, with 0 elements -$var3b: not set in universal scope - -$var3c: not set in local scope -$var3c: set in global scope, unexported, with 1 elements -$var3c[1]: length=10 value=|one string| -$var3c: not set in universal scope - - -#################### -# Prepending works -$var4a: not set in local scope -$var4a: set in global scope, unexported, with 6 elements -$var4a[1]: length=1 value=|e| -$var4a[2]: length=1 value=|f| -$var4a[3]: length=1 value=|d| -$var4a[4]: length=1 value=|a| -$var4a[5]: length=1 value=|b| -$var4a[6]: length=1 value=|c| -$var4a: not set in universal scope - -$var4b: not set in local scope -$var4b: set in global scope, unexported, with 0 elements -$var4b: not set in universal scope - -$var4c: not set in local scope -$var4c: set in global scope, unexported, with 1 elements -$var4c[1]: length=10 value=|one string| -$var4c: not set in universal scope - - -#################### -# Appending and prepending at same time works -$var5: not set in local scope -$var5: set in global scope, unexported, with 8 elements -$var5[1]: length=1 value=|0| -$var5[2]: length=1 value=|x| -$var5[3]: length=1 value=|0| -$var5[4]: length=3 value=|abc| -$var5[5]: length=3 value=|def| -$var5[6]: length=1 value=|0| -$var5[7]: length=1 value=|x| -$var5[8]: length=1 value=|0| -$var5: not set in universal scope - diff --git a/tests/test8.err b/tests/test8.err index 279b7f5aa..e69de29bb 100644 --- a/tests/test8.err +++ b/tests/test8.err @@ -1,21 +0,0 @@ - -#################### -# Test variable expand - -#################### -# Test variable set - -#################### -# Test using slices of command substitution - -#################### -# Test more - -#################### -# Verify that if statements swallow failure - -#################### -# Verify and/or behavior with if and while statements - -#################### -# Catch this corner case, which should produce an error diff --git a/tests/test8.in b/tests/test8.in index 2769765e6..97cc7d57d 100644 --- a/tests/test8.in +++ b/tests/test8.in @@ -1,6 +1,6 @@ # Test index ranges -logmsg Test variable expand +echo Test variable expand set n 10 set test (seq $n) echo $test[1..$n] # normal range @@ -9,26 +9,26 @@ echo $test[2..5 8..6] # several ranges echo $test[-1..-2] # range with negative limits echo $test[-1..1] # range with mixed limits -logmsg Test variable set +echo Test variable set set test1 $test set test1[-1..1] $test; echo $test1 set test1[1..$n] $test; echo $test1 set test1[$n..1] $test; echo $test1 set test1[2..4 -2..-4] $test1[4..2 -4..-2]; echo $test1 -logmsg Test using slices of command substitution +echo Test using slices of command substitution echo (seq 5)[-1..1] echo (seq $n)[3..5 -2..2] -logmsg Test more +echo Test more echo $test[(count $test)..1] echo $test[1..(count $test)] # See issue 1061 -logmsg Verify that if statements swallow failure +echo "Verify that if statements swallow failure" if false ; end ; echo $status -logmsg Verify and/or behavior with if and while statements +# Verify and/or behavior with if and while statements if false ; or true ; echo "success1" ; end if false ; and false ; echo "failure1" ; end while false ; and false ; or true ; echo "success2"; break ; end @@ -37,5 +37,5 @@ if false ; else if false ; and true ; else if false ; and false ; else if false; if false ; else if false ; and true ; else if false ; or false ; else if false; echo "failure 4"; end if false ; or true | false ; echo "failure5" ; end -logmsg Catch this corner case, which should produce an error +# Catch this corner case, which should produce an error if false ; or --help ; end diff --git a/tests/test8.out b/tests/test8.out index eb337f273..f21615f25 100644 --- a/tests/test8.out +++ b/tests/test8.out @@ -1,39 +1,23 @@ - -#################### -# Test variable expand +Test variable expand 1 2 3 4 5 6 7 8 9 10 10 9 8 7 6 5 4 3 2 1 2 3 4 5 8 7 6 10 9 10 9 8 7 6 5 4 3 2 1 - -#################### -# Test variable set +Test variable set 10 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 10 9 8 7 6 5 4 3 2 1 10 7 8 9 6 5 2 3 4 1 - -#################### -# Test using slices of command substitution +Test using slices of command substitution 5 4 3 2 1 3 4 5 9 8 7 6 5 4 3 2 - -#################### -# Test more +Test more 10 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 - -#################### -# Verify that if statements swallow failure +Verify that if statements swallow failure 0 - -#################### -# Verify and/or behavior with if and while statements success1 success2 success3 success4 - -#################### -# Catch this corner case, which should produce an error From 10498059e4d82418a83cf48ab5c58172038a0958 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:36:39 -0700 Subject: [PATCH 76/79] Revert "document command substitution data limit" This reverts commit 2bbcc5cbc80966d80576d1c30c14442ef6a6268e. --- doc_src/index.hdr.in | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/doc_src/index.hdr.in b/doc_src/index.hdr.in index 8994a0cb4..51f021d64 100644 --- a/doc_src/index.hdr.in +++ b/doc_src/index.hdr.in @@ -470,12 +470,10 @@ end The output of a series of commands can be used as the parameters to another command. If a parameter contains a set of parenthesis, the text enclosed by the parenthesis will be interpreted as a list of commands. On expansion, this list is executed, and substituted by the output. If the output is more than one line long, each line will be expanded to a new parameter. Setting `IFS` to the empty string will disable line splitting. -The exit status of the last run command substitution is available in the status variable if the substitution occurs in the context of a `set` command. +The exit status of the last run command substitution is available in the status variable. Only part of the output can be used, see index range expansion for details. -Fish has a default limit of 10 MiB on the amount of data a command substitution can output. If the limit is exceeded the entire command, not just the substitution, is failed and `$status` is set to 122. You can modify the limit by setting the `FISH_READ_BYTE_LIMIT` variable at any time including in the environment before fish starts running. If you set it to zero then no limit is imposed. This is a safety mechanism to keep the shell from consuming an too much memory if a command outputs an unreasonable amount of data. Note that this limit also affects how much data the `read` command will process. - Examples: \fish From e6fbb93d31a06c73d112299f11dade5551d33f58 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:39:13 -0700 Subject: [PATCH 77/79] Revert "implement limits on command substitution output" This reverts commit 4197420f3968093ec12075f1eac06ed1570a8e30. It was meant for the major branch. --- CHANGELOG.md | 2 +- src/exec.cpp | 41 ++++++------- src/exec.h | 5 +- src/expand.cpp | 101 +++++++++++++++------------------ src/fish_tests.cpp | 6 +- src/io.cpp | 5 +- src/io.h | 99 +++++--------------------------- src/parse_execution.cpp | 2 +- src/parser.cpp | 1 + tests/test8.in | 4 +- tests/test8.out | 2 +- tests/test_cmdsub.err | 34 ----------- tests/test_cmdsub.in | 33 ----------- tests/test_cmdsub.out | 29 ---------- tests/test_functions/show.fish | 18 ++++++ 15 files changed, 105 insertions(+), 277 deletions(-) delete mode 100644 tests/test_cmdsub.err delete mode 100644 tests/test_cmdsub.in delete mode 100644 tests/test_cmdsub.out create mode 100644 tests/test_functions/show.fish diff --git a/CHANGELOG.md b/CHANGELOG.md index 05863074a..5c79b768a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ This section is for changes merged to the `major` branch that are not also merged to 2.7. ## Deprecations -- The `IFS` variable is deprecated and will be removed in fish 4.0 (#4156). +- The `IFS` variable (#4156). ## Notable non-backward compatible changes - `.` command no longer exists -- use `source` (#4294). diff --git a/src/exec.cpp b/src/exec.cpp index 44f41255c..462c2448e 100644 --- a/src/exec.cpp +++ b/src/exec.cpp @@ -382,15 +382,13 @@ void exec_job(parser_t &parser, job_t *j) { // Verify that all IO_BUFFERs are output. We used to support a (single, hacked-in) magical input // IO_BUFFER used by fish_pager, but now the claim is that there are no more clients and it is // removed. This assertion double-checks that. - size_t stdout_read_limit = 0; const io_chain_t all_ios = j->all_io_redirections(); for (size_t idx = 0; idx < all_ios.size(); idx++) { const shared_ptr &io = all_ios.at(idx); if ((io->io_mode == IO_BUFFER)) { io_buffer_t *io_buffer = static_cast(io.get()); - assert(!io_buffer->is_input); - stdout_read_limit = io_buffer->get_buffer_limit(); + assert(!io_buffer->is_input); //!OCLINT(multiple unary operator) } } @@ -607,7 +605,7 @@ void exec_job(parser_t &parser, job_t *j) { shared_ptr block_output_io_buffer; // This is the io_streams we pass to internal builtins. - std::unique_ptr builtin_io_streams(new io_streams_t(stdout_read_limit)); + std::unique_ptr builtin_io_streams; switch (p->type) { case INTERNAL_FUNCTION: { @@ -756,6 +754,7 @@ void exec_job(parser_t &parser, job_t *j) { stdin_is_directly_redirected = stdin_io && stdin_io->io_mode != IO_CLOSE; } + builtin_io_streams.reset(new io_streams_t()); builtin_io_streams->stdin_fd = local_builtin_stdin; builtin_io_streams->out_is_redirected = has_fd(process_net_io_chain, STDOUT_FILENO); @@ -883,9 +882,8 @@ void exec_job(parser_t &parser, job_t *j) { const bool stdout_is_to_buffer = stdout_io && stdout_io->io_mode == IO_BUFFER; const bool no_stdout_output = stdout_buffer.empty(); const bool no_stderr_output = stderr_buffer.empty(); - const bool stdout_discarded = builtin_io_streams->out.output_discarded(); - if (!stdout_discarded && no_stdout_output && no_stderr_output) { + if (no_stdout_output && no_stderr_output) { // The builtin produced no output and is not inside of a pipeline. No // need to fork or even output anything. debug(3, L"Skipping fork: no output for internal builtin '%ls'", @@ -899,15 +897,13 @@ void exec_job(parser_t &parser, job_t *j) { p->argv0()); io_buffer_t *io_buffer = static_cast(stdout_io.get()); - if (stdout_discarded) { - io_buffer->set_discard(); - } else { - const std::string res = wcs2string(builtin_io_streams->out.buffer()); - io_buffer->out_buffer_append(res.data(), res.size()); - } + const std::string res = wcs2string(builtin_io_streams->out.buffer()); + + io_buffer->out_buffer_append(res.data(), res.size()); fork_was_skipped = true; } else if (stdout_io.get() == NULL && stderr_io.get() == NULL) { - // We are writing to normal stdout and stderr. Just do it - no need to fork. + // We are writing to normal stdout and stderr. Just do it - no need to + // fork. debug(3, L"Skipping fork: ordinary output for internal builtin '%ls'", p->argv0()); const std::string outbuff = wcs2string(stdout_buffer); @@ -919,7 +915,6 @@ void exec_job(parser_t &parser, job_t *j) { debug(0, "!builtin_io_done and errno != EPIPE"); show_stackframe(L'E'); } - if (stdout_discarded) p->status = STATUS_READ_TOO_MUCH; fork_was_skipped = true; } } @@ -1104,8 +1099,8 @@ void exec_job(parser_t &parser, job_t *j) { } } -static int exec_subshell_internal(const wcstring &cmd, wcstring_list_t *lst, bool apply_exit_status, - bool is_subcmd) { +static int exec_subshell_internal(const wcstring &cmd, wcstring_list_t *lst, + bool apply_exit_status) { ASSERT_IS_MAIN_THREAD(); bool prev_subshell = is_subshell; const int prev_status = proc_get_last_status(); @@ -1121,8 +1116,7 @@ static int exec_subshell_internal(const wcstring &cmd, wcstring_list_t *lst, boo // IO buffer creation may fail (e.g. if we have too many open files to make a pipe), so this may // be null. - const shared_ptr io_buffer( - io_buffer_t::create(STDOUT_FILENO, io_chain_t(), is_subcmd ? read_byte_limit : 0)); + const shared_ptr io_buffer(io_buffer_t::create(STDOUT_FILENO, io_chain_t())); if (io_buffer.get() != NULL) { parser_t &parser = parser_t::principal_parser(); if (parser.eval(cmd, io_chain_t(io_buffer), SUBST) == 0) { @@ -1132,8 +1126,6 @@ static int exec_subshell_internal(const wcstring &cmd, wcstring_list_t *lst, boo io_buffer->read(); } - if (io_buffer->output_discarded()) subcommand_status = STATUS_READ_TOO_MUCH; - // If the caller asked us to preserve the exit status, restore the old status. Otherwise set the // status of the subcommand. proc_set_last_status(apply_exit_status ? subcommand_status : prev_status); @@ -1174,13 +1166,12 @@ static int exec_subshell_internal(const wcstring &cmd, wcstring_list_t *lst, boo return subcommand_status; } -int exec_subshell(const wcstring &cmd, std::vector &outputs, bool apply_exit_status, - bool is_subcmd) { +int exec_subshell(const wcstring &cmd, std::vector &outputs, bool apply_exit_status) { ASSERT_IS_MAIN_THREAD(); - return exec_subshell_internal(cmd, &outputs, apply_exit_status, is_subcmd); + return exec_subshell_internal(cmd, &outputs, apply_exit_status); } -int exec_subshell(const wcstring &cmd, bool apply_exit_status, bool is_subcmd) { +int exec_subshell(const wcstring &cmd, bool apply_exit_status) { ASSERT_IS_MAIN_THREAD(); - return exec_subshell_internal(cmd, NULL, apply_exit_status, is_subcmd); + return exec_subshell_internal(cmd, NULL, apply_exit_status); } diff --git a/src/exec.h b/src/exec.h index 27a5110c8..7ce80fb6f 100644 --- a/src/exec.h +++ b/src/exec.h @@ -35,9 +35,8 @@ void exec_job(parser_t &parser, job_t *j); /// \param outputs The list to insert output into. /// /// \return the status of the last job to exit, or -1 if en error was encountered. -int exec_subshell(const wcstring &cmd, std::vector &outputs, bool preserve_exit_status, - bool is_subcmd = false); -int exec_subshell(const wcstring &cmd, bool preserve_exit_status, bool is_subcmd = false); +int exec_subshell(const wcstring &cmd, std::vector &outputs, bool preserve_exit_status); +int exec_subshell(const wcstring &cmd, bool preserve_exit_status); /// Loops over close until the syscall was run without being interrupted. void exec_close(int fd); diff --git a/src/expand.cpp b/src/expand.cpp index 79f4cec7d..347f0c3c3 100644 --- a/src/expand.cpp +++ b/src/expand.cpp @@ -102,43 +102,37 @@ static bool expand_is_clean(const wcstring &in) { /// Append a syntax error to the given error list. static void append_syntax_error(parse_error_list_t *errors, size_t source_start, const wchar_t *fmt, ...) { - if (!errors) return; + if (errors != NULL) { + parse_error_t error; + error.source_start = source_start; + error.source_length = 0; + error.code = parse_error_syntax; - parse_error_t error; - error.source_start = source_start; - error.source_length = 0; - error.code = parse_error_syntax; + va_list va; + va_start(va, fmt); + error.text = vformat_string(fmt, va); + va_end(va); - va_list va; - va_start(va, fmt); - error.text = vformat_string(fmt, va); - va_end(va); - - errors->push_back(error); + errors->push_back(error); + } } -/// Append a cmdsub error to the given error list. But only do so if the error hasn't already been -/// recorded. This is needed because command substitution is a recursive process and some errors -/// could consequently be recorded more than once. +/// Append a cmdsub error to the given error list. static void append_cmdsub_error(parse_error_list_t *errors, size_t source_start, const wchar_t *fmt, ...) { - if (!errors) return; + if (errors != NULL) { + parse_error_t error; + error.source_start = source_start; + error.source_length = 0; + error.code = parse_error_cmdsubst; - parse_error_t error; - error.source_start = source_start; - error.source_length = 0; - error.code = parse_error_cmdsubst; + va_list va; + va_start(va, fmt); + error.text = vformat_string(fmt, va); + va_end(va); - va_list va; - va_start(va, fmt); - error.text = vformat_string(fmt, va); - va_end(va); - - for (auto it : *errors) { - if (error.text == it.text) return; + errors->push_back(error); } - - errors->push_back(error); } /// Return the environment variable value for the string starting at \c in. @@ -1030,23 +1024,24 @@ static expand_error_t expand_brackets(const wcstring &instr, expand_flags_t flag } /// Perform cmdsubst expansion. -static bool expand_cmdsubst(const wcstring &input, std::vector *out_list, - parse_error_list_t *errors) { - wchar_t *paren_begin = nullptr, *paren_end = nullptr; - wchar_t *tail_begin = nullptr; +static int expand_cmdsubst(const wcstring &input, std::vector *out_list, + parse_error_list_t *errors) { + wchar_t *paran_begin = 0, *paran_end = 0; + std::vector sub_res; size_t i, j; + wchar_t *tail_begin = 0; const wchar_t *const in = input.c_str(); int parse_ret; - switch (parse_ret = parse_util_locate_cmdsubst(in, &paren_begin, &paren_end, false)) { + switch (parse_ret = parse_util_locate_cmdsubst(in, ¶n_begin, ¶n_end, false)) { case -1: { append_syntax_error(errors, SOURCE_LOCATION_UNKNOWN, L"Mismatched parenthesis"); - return false; + return 0; } case 0: { append_completion(out_list, input); - return true; + return 1; } case 1: { break; @@ -1057,22 +1052,15 @@ static bool expand_cmdsubst(const wcstring &input, std::vector *ou } } - wcstring_list_t sub_res; - const wcstring subcmd(paren_begin + 1, paren_end - paren_begin - 1); - if (exec_subshell(subcmd, sub_res, true /* apply_exit_status */, true /* is_subcmd */) == -1) { + const wcstring subcmd(paran_begin + 1, paran_end - paran_begin - 1); + + if (exec_subshell(subcmd, sub_res, true /* do apply exit status */) == -1) { append_cmdsub_error(errors, SOURCE_LOCATION_UNKNOWN, L"Unknown error while evaulating command substitution"); - return false; + return 0; } - if (proc_get_last_status() == STATUS_READ_TOO_MUCH) { - append_cmdsub_error( - errors, in - paren_begin, - _(L"Too much data emitted by command substitution so it was discarded\n")); - return false; - } - - tail_begin = paren_end + 1; + tail_begin = paran_end + 1; if (*tail_begin == L'[') { std::vector slice_idx; std::vector slice_source_positions; @@ -1084,7 +1072,7 @@ static bool expand_cmdsubst(const wcstring &input, std::vector *ou parse_slice(slice_begin, &slice_end, slice_idx, slice_source_positions, sub_res.size()); if (bad_pos != 0) { append_syntax_error(errors, slice_begin - in + bad_pos, L"Invalid index value"); - return false; + return 0; } wcstring_list_t sub_res2; @@ -1122,7 +1110,7 @@ static bool expand_cmdsubst(const wcstring &input, std::vector *ou const wcstring &tail_item = tail_expand.at(j).completion; // sb_append_substring( &whole_item, in, len1 ); - whole_item.append(in, paren_begin - in); + whole_item.append(in, paran_begin - in); // sb_append_char( &whole_item, INTERNAL_SEPARATOR ); whole_item.push_back(INTERNAL_SEPARATOR); @@ -1141,8 +1129,7 @@ static bool expand_cmdsubst(const wcstring &input, std::vector *ou } } - if (proc_get_last_status() == STATUS_READ_TOO_MUCH) return false; - return true; + return 1; } // Given that input[0] is HOME_DIRECTORY or tilde (ugh), return the user's name. Return the empty @@ -1317,6 +1304,7 @@ typedef expand_error_t (*expand_stage_t)(const wcstring &input, //!OCL static expand_error_t expand_stage_cmdsubst(const wcstring &input, std::vector *out, expand_flags_t flags, parse_error_list_t *errors) { + expand_error_t result = EXPAND_OK; if (EXPAND_SKIP_CMDSUBST & flags) { wchar_t *begin, *end; if (parse_util_locate_cmdsubst(input.c_str(), &begin, &end, true) == 0) { @@ -1324,14 +1312,15 @@ static expand_error_t expand_stage_cmdsubst(const wcstring &input, std::vector *out, diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 09e57e091..0702efb9c 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -2013,7 +2013,7 @@ static bool run_one_test_test(int expected, wcstring_list_t &lst, bool bracket) i++; } argv[i + 1] = NULL; - io_streams_t streams(0); + io_streams_t streams; int result = builtin_test(parser, streams, argv); delete[] argv; return expected == result; @@ -2040,7 +2040,7 @@ static bool run_test_test(int expected, const wcstring &str) { static void test_test_brackets() { // Ensure [ knows it needs a ]. parser_t parser; - io_streams_t streams(0); + io_streams_t streams; const wchar_t *argv1[] = {L"[", L"foo", NULL}; do_test(builtin_test(parser, streams, (wchar_t **)argv1) != 0); @@ -3930,7 +3930,7 @@ int builtin_string(parser_t &parser, io_streams_t &streams, wchar_t **argv); static void run_one_string_test(const wchar_t **argv, int expected_rc, const wchar_t *expected_out) { parser_t parser; - io_streams_t streams(0); + io_streams_t streams; streams.stdin_is_directly_redirected = false; // read from argv instead of stdin int rc = builtin_string(parser, streams, const_cast(argv)); wcstring args; diff --git a/src/io.cpp b/src/io.cpp index b36a8d500..f5b315430 100644 --- a/src/io.cpp +++ b/src/io.cpp @@ -75,11 +75,10 @@ bool io_buffer_t::avoid_conflicts_with_io_chain(const io_chain_t &ios) { return result; } -shared_ptr io_buffer_t::create(int fd, const io_chain_t &conflicts, - size_t buffer_limit) { +shared_ptr io_buffer_t::create(int fd, const io_chain_t &conflicts) { bool success = true; assert(fd >= 0); - shared_ptr buffer_redirect(new io_buffer_t(fd, buffer_limit)); + shared_ptr buffer_redirect(new io_buffer_t(fd)); if (exec_pipe(buffer_redirect->pipe_fd) == -1) { debug(1, PIPE_ERROR); diff --git a/src/io.h b/src/io.h index 4a24edee7..a2340362e 100644 --- a/src/io.h +++ b/src/io.h @@ -20,7 +20,6 @@ using std::tr1::shared_ptr; #endif #include "common.h" -#include "env.h" /// Describes what type of IO operation an io_data_t represents. enum io_mode_t { IO_FILE, IO_PIPE, IO_FD, IO_BUFFER, IO_CLOSE }; @@ -100,18 +99,10 @@ class io_pipe_t : public io_data_t { class io_chain_t; class io_buffer_t : public io_pipe_t { private: - /// True if we're discarding input. - bool discard; - /// Limit on how much data we'll buffer. Zero means no limit. - size_t buffer_limit; /// Buffer to save output in. std::vector out_buffer; - explicit io_buffer_t(int f, size_t limit) - : io_pipe_t(IO_BUFFER, f, false /* not input */), - discard(false), - buffer_limit(limit), - out_buffer() {} + explicit io_buffer_t(int f) : io_pipe_t(IO_BUFFER, f, false /* not input */), out_buffer() {} public: virtual void print() const; @@ -120,12 +111,6 @@ class io_buffer_t : public io_pipe_t { /// Function to append to the buffer. void out_buffer_append(const char *ptr, size_t count) { - if (discard) return; - if (buffer_limit && out_buffer.size() + count > buffer_limit) { - discard = true; - out_buffer.clear(); - return; - } out_buffer.insert(out_buffer.end(), ptr, ptr + count); } @@ -137,19 +122,6 @@ class io_buffer_t : public io_pipe_t { /// Function to get the size of the buffer. size_t out_buffer_size(void) const { return out_buffer.size(); } - /// Function that returns true if we discarded the input because there was too much data. - bool output_discarded(void) { return discard; } - - /// Function to explicitly put the object in discard mode. Meant to be used when moving - /// the results from an output_stream_t to an io_buffer_t. - void set_discard(void) { - discard = true; - out_buffer.clear(); - } - - /// This is used to transfer the buffer limit for this object to a output_stream_t object. - size_t get_buffer_limit(void) { return buffer_limit; } - /// Ensures that the pipes do not conflict with any fd redirections in the chain. bool avoid_conflicts_with_io_chain(const io_chain_t &ios); @@ -162,8 +134,7 @@ class io_buffer_t : public io_pipe_t { /// \param fd the fd that will be mapped in the child process, typically STDOUT_FILENO /// \param conflicts A set of IO redirections. The function ensures that any pipe it makes does /// not conflict with an fd redirection in this list. - static shared_ptr create(int fd, const io_chain_t &conflicts, - size_t buffer_limit = 0); + static shared_ptr create(int fd, const io_chain_t &conflicts); }; class io_chain_t : public std::vector > { @@ -193,79 +164,37 @@ bool pipe_avoid_conflicts_with_io_chain(int fds[2], const io_chain_t &ios); /// Class representing the output that a builtin can generate. class output_stream_t { private: - /// Limit on how much data we'll buffer. Zero means no limit. - size_t buffer_limit; - /// True if we're discarding input. - bool discard; // No copying. output_stream_t(const output_stream_t &s); void operator=(const output_stream_t &s); wcstring buffer_; - void check_for_overflow() { - if (buffer_limit && buffer_.size() > buffer_limit) { - discard = true; - buffer_.clear(); - } - } - public: - output_stream_t(size_t buffer_limit_) : buffer_limit(buffer_limit_), discard(false) {} + output_stream_t() {} -#if 0 - void set_buffer_limit(size_t buffer_limit_) { buffer_limit = buffer_limit_; } -#endif + void append(const wcstring &s) { this->buffer_.append(s); } - void append(const wcstring &s) { - if (discard) return; - buffer_.append(s); - check_for_overflow(); - } + void append(const wchar_t *s) { this->buffer_.append(s); } - void append(const wchar_t *s) { - if (discard) return; - buffer_.append(s); - check_for_overflow(); - } + void append(wchar_t s) { this->buffer_.push_back(s); } - void append(wchar_t s) { - if (discard) return; - buffer_.push_back(s); - check_for_overflow(); - } + void append(const wchar_t *s, size_t amt) { this->buffer_.append(s, amt); } - void append(const wchar_t *s, size_t amt) { - if (discard) return; - buffer_.append(s, amt); - check_for_overflow(); - } - - void push_back(wchar_t c) { - if (discard) return; - buffer_.push_back(c); - check_for_overflow(); - } + void push_back(wchar_t c) { this->buffer_.push_back(c); } void append_format(const wchar_t *format, ...) { - if (discard) return; va_list va; va_start(va, format); - ::append_formatv(buffer_, format, va); + ::append_formatv(this->buffer_, format, va); va_end(va); - check_for_overflow(); } void append_formatv(const wchar_t *format, va_list va_orig) { - if (discard) return; - ::append_formatv(buffer_, format, va_orig); - check_for_overflow(); + ::append_formatv(this->buffer_, format, va_orig); } - const wcstring &buffer() const { return buffer_; } - - /// Function that returns true if we discarded the input because there was too much data. - bool output_discarded(void) { return discard; } + const wcstring &buffer() const { return this->buffer_; } bool empty() const { return buffer_.empty(); } }; @@ -289,10 +218,8 @@ struct io_streams_t { // Actual IO redirections. This is only used by the source builtin. Unowned. const io_chain_t *io_chain; - io_streams_t(size_t read_limit) - : out(read_limit), - err(read_limit), - stdin_fd(-1), + io_streams_t() + : stdin_fd(-1), stdin_is_directly_redirected(false), out_is_redirected(false), err_is_redirected(false), diff --git a/src/parse_execution.cpp b/src/parse_execution.cpp index f21327569..958a95ebc 100644 --- a/src/parse_execution.cpp +++ b/src/parse_execution.cpp @@ -388,7 +388,7 @@ parse_execution_result_t parse_execution_context_t::run_function_statement( const wcstring contents_str = wcstring(this->src, contents_start, contents_end - contents_start); int definition_line_offset = this->line_offset_of_character_at_offset(contents_start); - io_streams_t streams(0); // no limit on the amount of output from builtin_function() + io_streams_t streams; int err = builtin_function(*parser, streams, argument_list, contents_str, definition_line_offset); proc_set_last_status(err); diff --git a/src/parser.cpp b/src/parser.cpp index ad9e07398..22859abf6 100644 --- a/src/parser.cpp +++ b/src/parser.cpp @@ -724,6 +724,7 @@ int parser_t::eval_block_node(node_offset_t node_idx, const io_chain_t &io, this->pop_block(scope_block); job_reap(0); // reap again + return result; } diff --git a/tests/test8.in b/tests/test8.in index 97cc7d57d..5e918fbe8 100644 --- a/tests/test8.in +++ b/tests/test8.in @@ -1,4 +1,4 @@ -# Test index ranges +# Test index ranges echo Test variable expand set n 10 @@ -16,7 +16,7 @@ set test1[1..$n] $test; echo $test1 set test1[$n..1] $test; echo $test1 set test1[2..4 -2..-4] $test1[4..2 -4..-2]; echo $test1 -echo Test using slices of command substitution +echo Test command substitution echo (seq 5)[-1..1] echo (seq $n)[3..5 -2..2] diff --git a/tests/test8.out b/tests/test8.out index f21615f25..1e9099f8e 100644 --- a/tests/test8.out +++ b/tests/test8.out @@ -9,7 +9,7 @@ Test variable set 1 2 3 4 5 6 7 8 9 10 10 9 8 7 6 5 4 3 2 1 10 7 8 9 6 5 2 3 4 1 -Test using slices of command substitution +Test command substitution 5 4 3 2 1 3 4 5 9 8 7 6 5 4 3 2 Test more diff --git a/tests/test_cmdsub.err b/tests/test_cmdsub.err deleted file mode 100644 index cc56e58f2..000000000 --- a/tests/test_cmdsub.err +++ /dev/null @@ -1,34 +0,0 @@ - -#################### -# Command sub just under the limit should succeed - -#################### -# Command sub at the limit should fail -fish: Too much data emitted by command substitution so it was discarded - -set b (string repeat -n 512 x) - ^ - -#################### -# Command sub over the limit should fail -fish: Too much data emitted by command substitution so it was discarded - -set -l x (string repeat -n $argv x) - ^ -in function 'subme' - called on standard input - with parameter list '513' - -in command substitution - called on standard input - - -#################### -# Make sure output from builtins outside of command substitution is not affected - -#################### -# Same builtin in a command substitution is affected -fish: Too much data emitted by command substitution so it was discarded - -echo this will fail (string repeat --max 513 b) to output anything - ^ diff --git a/tests/test_cmdsub.in b/tests/test_cmdsub.in deleted file mode 100644 index 29507a7f0..000000000 --- a/tests/test_cmdsub.in +++ /dev/null @@ -1,33 +0,0 @@ -# This tests various corner cases involving command substitution. Most -# importantly the limits on the amount of data we'll substitute. - -set FISH_READ_BYTE_LIMIT 512 - -function subme - set -l x (string repeat -n $argv x) - echo $x -end - -logmsg Command sub just under the limit should succeed -set a (subme 511) -set --show a - -logmsg Command sub at the limit should fail -set b (string repeat -n 512 x) -set saved_status $status -test $saved_status -eq 122 -or echo expected status 122, saw $saved_status >&2 -set --show b - -logmsg Command sub over the limit should fail -set c (subme 513) -set --show c - -logmsg Make sure output from builtins outside of command substitution is not affected -string repeat --max 513 a - -logmsg Same builtin in a command substitution is affected -echo this will fail (string repeat --max 513 b) to output anything -set saved_status $status -test $saved_status -eq 122 -or echo expected status 122, saw $saved_status >&2 diff --git a/tests/test_cmdsub.out b/tests/test_cmdsub.out deleted file mode 100644 index 26011ce4b..000000000 --- a/tests/test_cmdsub.out +++ /dev/null @@ -1,29 +0,0 @@ - -#################### -# Command sub just under the limit should succeed -$a: not set in local scope -$a: set in global scope, unexported, with 1 elements -$a[1]: length=511 value=|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| -$a: not set in universal scope - - -#################### -# Command sub at the limit should fail -$b: not set in local scope -$b: not set in global scope -$b: not set in universal scope - - -#################### -# Command sub over the limit should fail -$c: not set in local scope -$c: set in global scope, unexported, with 0 elements -$c: not set in universal scope - - -#################### -# Make sure output from builtins outside of command substitution is not affected -aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa - -#################### -# Same builtin in a command substitution is affected diff --git a/tests/test_functions/show.fish b/tests/test_functions/show.fish new file mode 100644 index 000000000..bbf6c04cd --- /dev/null +++ b/tests/test_functions/show.fish @@ -0,0 +1,18 @@ +# Show information about the named var(s) passed to us. +function show --no-scope-shadowing + for v in $argv + if set -q $v + set -l c (count $$v) + printf '$%s count=%d\n' $v $c + # This test is to work around a bogosity of the BSD `seq` command. If called with + # `seq 0` it emits `1`, `0`. Whereas that GNU version outputs nothing. + if test $c -gt 0 + for i in (seq $c) + printf '$%s[%d]=|%s|\n' $v $i $$v[1][$i] + end + end + else + echo \$$v is not set + end + end +end From ef25d8c76d7fdfaeef9f628cc3520ab0b30f8ce7 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 15:44:31 -0700 Subject: [PATCH 78/79] Revert "rewrite `abbr` function" This reverts commit 17dff8c569a4bbbe88d23c77e043574d15804093. It was meant for the major branch. --- doc_src/abbr.txt | 39 +- .../functions/__fish_config_interactive.fish | 9 - share/functions/abbr.fish | 397 +++++++++--------- share/functions/abbr_old.fish | 206 --------- src/env.cpp | 34 +- src/expand.cpp | 56 ++- src/expand.h | 3 +- src/fish_tests.cpp | 20 +- tests/abbr.err | 59 +-- tests/abbr.in | 52 +-- tests/abbr.out | 65 +-- tests/functions.in | 2 +- 12 files changed, 322 insertions(+), 620 deletions(-) delete mode 100644 share/functions/abbr_old.fish diff --git a/doc_src/abbr.txt b/doc_src/abbr.txt index 18e209c2a..c1fd717ee 100644 --- a/doc_src/abbr.txt +++ b/doc_src/abbr.txt @@ -2,57 +2,54 @@ \subsection abbr-synopsis Synopsis \fish{synopsis} -abbr --add [SCOPE] WORD EXPANSION -abbr --erase word -abbr --rename [SCOPE] OLD_WORD NEW_WORD +abbr --add word phrase... +abbr --rename word new_word abbr --show abbr --list +abbr --erase word \endfish \subsection abbr-description Description -Abbreviations are user-defined character sequences or words that are replaced with longer phrases after they are entered. For example, a frequently-run command such as `git checkout` can be abbreviated to `gco`. After entering `gco` and pressing @key{Space} or @key{Enter}, the full text `git checkout` will appear in the command line. The `abbr` command manipulates those abbreviations. +`abbr` manipulates the list of abbreviations that fish will expand. -Each abbreviation is stored in its own global or universal variable. The name consists of the prefix `_fish_abbr_` followed by the WORD after being transformed by `string escape style=var`. The WORD cannot contain a space but all other characters are legal. +Abbreviations are user-defined character sequences or words that are replaced with longer phrases after they are entered. For example, a frequently-run command such as `git checkout` can be abbreviated to `gco`. After entering `gco` and pressing @key{Space} or @key{Enter}, the full text `git checkout` will appear in the command line. -Defining an abbreviation with global scope is slightly faster than universal scope (which is the default). But in general you'll only want to use the global scope when defining abbreviations in a startup script like `~/.config/fish/config.fish` like this: +Abbreviations are stored in a variable named `fish_user_abbreviations`. This is automatically created as a universal variable the first time an abbreviation is created. If you want your abbreviations to be private to a particular fish session you can put the following in your *~/.config/fish/config.fish* file before you define your first abbrevation: \fish if status --is-interactive - abbr --add --global first 'echo my first abbreviation' - abbr --add --global second 'echo my second abbreviation' - abbr --add --global gco git checkout + set -g fish_user_abbreviations + abbr --add first 'echo my first abbreviation' + abbr --add second 'echo my second abbreviation' # etcetera end \endfish -You can create abbreviations interactively and they will be visible to other fish sessions if you use the `-U` or `--universal` flag or don't explicitly specify the scope and the abbreviation isn't already defined with global scope. If you want it to be visible only to the current shell use the `-g` or `--global` flag. +You can create abbreviations directly on the command line and they will be saved automatically and made visible to other fish sessions if `fish_user_abbreviations` is a universal variable. If you keep the variable as universal, `abbr --add` statements in config.fish will do nothing but slow down startup slightly. \subsection abbr-options Options -The following options are available: +The following parameters are available: -- `-a WORD EXPANSION` or `--add WORD EXPANSION` Adds a new abbreviation, causing WORD to be expanded to PHRASE. You can optionally specify `-g` or `--global` to avoid the overhead of universal variables at the expense of not having the definition being immediately visible to other fish shells that are already running. If you don't specify global scope it default to universal. For clarity you can also specify `-U` or `--universal`. +- `-a WORD PHRASE` or `--add WORD PHRASE` Adds a new abbreviation, causing WORD to be expanded to PHRASE. -- `-r OLD_WORD NEW_WORD` or `--rename OLD_WORD NEW_WORD` Renames an abbreviation, from OLD_WORD to NEW_WORD. +- `-r WORD NEW_WORD` or `--rename WORD NEW_WORD` Renames an abbreviation, from WORD to NEW_WORD. -- `-s` or `--show` Show all abbreviations in a manner suitable for export and import. +- `-s` or `--show` Show all abbreviated words and their expanded phrases in a manner suitable for export and import. - `-l` or `--list` Lists all abbreviated words. - `-e WORD` or `--erase WORD` Erase the abbreviation WORD. +Note: fish version 2.1 supported `-a WORD=PHRASE`. This syntax is now deprecated but will still be converted. + \subsection abbr-example Examples \fish -abbr -a -g gco git checkout +abbr -a gco git checkout \endfish -Add a new abbreviation where `gco` will be replaced with `git checkout` global to the current shell. This abbreviation will not be automatically visible to other shells unless the same command is run in those shells (such as when executing the commands in config.fish). - -\fish -abbr -a -U l less -\endfish -Add a new abbreviation where `l` will be replaced with `less` univeral so all shells. Note that you omit the `-U` since it is the default. +Add a new abbreviation where `gco` will be replaced with `git checkout`. \fish abbr -r gco gch diff --git a/share/functions/__fish_config_interactive.fish b/share/functions/__fish_config_interactive.fish index 5d8c92ff7..437605022 100644 --- a/share/functions/__fish_config_interactive.fish +++ b/share/functions/__fish_config_interactive.fish @@ -4,15 +4,6 @@ # This function is called by the __fish_on_interactive function, which is defined in config.fish. # function __fish_config_interactive -d "Initializations that should be performed when entering interactive mode" - if not set -q __fish_init_3_x - # Perform transitions relevant to going from fish 2.x to 3.x. - - # Migrate old universal abbreviations to the new scheme. - abbr_old | source - - set -U __fish_init_3_x - end - # Make sure this function is only run once. if set -q __fish_config_interactive_done return diff --git a/share/functions/abbr.fish b/share/functions/abbr.fish index 508965fd6..23a2d08e3 100644 --- a/share/functions/abbr.fish +++ b/share/functions/abbr.fish @@ -1,201 +1,206 @@ -function abbr --description "Manage abbreviations using new fish 3.0 scheme." - set -l options --stop-nonopt --exclusive 'a,r,e,l,s' --exclusive 'g,U' - set options $options 'h/help' 'a/add' 'r/rename' 'e/erase' 'l/list' 's/show' - set options $options 'g/global' 'U/universal' +function abbr --description "Manage abbreviations" + # parse arguments + set -l mode + set -l mode_flag # the flag that was specified, for better errors + set -l mode_arg + set -l needs_arg no + while set -q argv[1] + set -l new_mode + switch $argv[1] + case '-h' '--help' + __fish_print_help abbr + return 0 + case '-a' '--add' + set new_mode add + set needs_arg multi + case '-r' '--rename' + set new_mode rename + set needs_arg double + case '-e' '--erase' + set new_mode erase + set needs_arg single + case '-l' '--list' + set new_mode list + case '-s' '--show' + set new_mode show + case '--' + set -e argv[1] + break + case '-*' + printf ( _ "%s: invalid option -- %s\n" ) abbr $argv[1] >&2 + return 1 + case '*' + break + end + if test -n "$mode" -a -n "$new_mode" + # we're trying to set two different modes + printf ( _ "%s: %s cannot be specified along with %s\n" ) abbr $argv[1] $mode_flag >&2 + return 1 + end + set mode $new_mode + set mode_flag $argv[1] + set -e argv[1] + end - argparse -n abbr $options -- $argv - or return + # If run with no options, treat it like --add if we have an argument, or + # --show if we do not have an argument + if not set -q mode[1] + if set -q argv[1] + set mode add + set needs_arg multi + else + set mode show + end + end - if set -q _flag_help - __fish_print_help abbr + if test $needs_arg = single + set mode_arg $argv[1] + set needs_arg no + set -e argv[1] + else if test $needs_arg = double + # Pull the two parameters from argv. + # * leave argv non-empty, if there are more than two arguments + # * leave needs_arg set to double if there is not enough arguments + if set -q argv[1] + set param1 $argv[1] + set -e argv[1] + if set -q argv[1] + set param2 $argv[1] + set needs_arg no + set -e argv[1] + end + end + else if test $needs_arg = multi + set mode_arg $argv + set needs_arg no + set -e argv + end + if test $needs_arg != no + printf ( _ "%s: option requires an argument -- %s\n" ) abbr $mode_flag >&2 + return 1 + end + + # none of our modes want any excess arguments + if set -q argv[1] + printf ( _ "%s: Unexpected argument -- %s\n" ) abbr $argv[1] >&2 + return 1 + end + + switch $mode + case 'add' + # Convert from old "key=value" syntax + # TODO: This should be removed later + if not set -q mode_arg[2] + and string match -qr '^[^ ]+=' -- $mode_arg + set mode_arg (string split "=" -- $mode_arg) + end + + # Bail out early if the exact abbr is already in + set -q fish_user_abbreviations + and contains -- "$mode_arg" $fish_user_abbreviations + and return 0 + + set -l key $mode_arg[1] + set -e mode_arg[1] + set -l value "$mode_arg" + # Because we later store "$key $value", there can't be any spaces in the key + if string match -q "* *" -- $key + printf ( _ "%s: abbreviation cannot have spaces in the key\n" ) abbr >&2 + return 1 + end + if test -z "$value" + printf ( _ "%s: abbreviation must have a value\n" ) abbr >&2 + return 1 + end + if set -l idx (__fish_abbr_get_by_key $key) + # erase the existing abbreviation + set -e fish_user_abbreviations[$idx] + end + if not set -q fish_user_abbreviations + # initialize as a universal variable, so we can skip the -U later + # and therefore work properly if someone sets this as a global variable + set -U fish_user_abbreviations + end + set fish_user_abbreviations $fish_user_abbreviations "$key $value" + return 0 + + case 'rename' + set -l old_name $param1 + set -l new_name $param2 + + # if the target name already exists, throw an error + if set -l idx (__fish_abbr_get_by_key $new_name) + printf ( _ "%s: abbreviation '%s' already exists, cannot rename\n" ) abbr $new_name >&2 + return 2 + end + + # Because we later store "$key $value", there can't be any spaces in the key + if string match -q "* *" -- $new_name + printf ( _ "%s: abbreviation cannot have spaces in the key\n" ) abbr >&2 + return 1 + end + + set -l idx (__fish_abbr_get_by_key $old_name) + or begin + printf ( _ "%s: no such abbreviation '%s'\n" ) abbr $old_name >&2 + return 2 + end + + set -l value (string split " " -m 1 -- $fish_user_abbreviations[$idx])[2] + set fish_user_abbreviations[$idx] "$new_name $value" + return 0 + + case 'erase' + if set -l idx (__fish_abbr_get_by_key $mode_arg) + set -e fish_user_abbreviations[$idx] + return 0 + else + printf ( _ "%s: no such abbreviation '%s'\n" ) abbr $mode_arg >&2 + return 2 + end + + case 'show' + for i in $fish_user_abbreviations + set -l opt_double_dash + set -l kv (string split " " -m 1 -- $i) + set -l key $kv[1] + set -l value $kv[2] + + # Check to see if either key or value has a leading dash + # If so, we need to write -- + string match -q -- '-*' $key $value + and set opt_double_dash '--' + echo abbr $opt_double_dash (string escape -- $key $value) + end + return 0 + + case 'list' + for i in $fish_user_abbreviations + set -l key (string split " " -m 1 -- $i)[1] + printf "%s\n" $key + end + return 0 + end +end + +function __fish_abbr_get_by_key + if not set -q argv[1] + echo "__fish_abbr_get_by_key: expected one argument, got none" >&2 + return 2 + end + + set -q fish_user_abbreviations + or return 1 + + # Going through all entries is still quicker than calling `seq` + set -l keys + for kv in $fish_user_abbreviations + # If this does not match, we have screwed up before and the error should be reported + set keys $keys (string split " " -m 1 -- $kv)[1] + end + if set -l idx (contains -i -- $argv[1] $keys) + echo $idx return 0 end - - # If run with no options, treat it like --add if we have arguments, or - # --show if we do not have any arguments. - set -l _flag_add - set -l _flag_show - if not set -q _flag_add[1] - and not set -q _flag_rename[1] - and not set -q _flag_erase[1] - and not set -q _flag_list[1] - and not set -q _flag_show[1] - if set -q argv[1] - set _flag_add --add - else - set _flag_show --show - end - end - - set -l abbr_scope - if set -q _flag_global - set abbr_scope --global - else if set -q _flag_universal - set abbr_scope --universal - end - - if set -q _flag_add[1] - __fish_abbr_add $argv - return - else if set -q _flag_erase[1] - __fish_abbr_erase $argv - return - else if set -q _flag_rename[1] - __fish_abbr_rename $argv - return - else if set -q _flag_list[1] - __fish_abbr_list $argv - return - else if set -q _flag_show[1] - __fish_abbr_show $argv - return - else - printf ( _ "%s: Could not figure out what to do!\n" ) abbr >&2 - return 127 - end -end - -function __fish_abbr_add --no-scope-shadowing - if not set -q argv[2] - printf ( _ "%s %s: Requires at least two arguments\n" ) abbr --add >&2 - return 1 - end - - # Because of the way abbreviations are expanded there can't be any spaces in the key. - set -l abbr_name $argv[1] - set -l escaped_abbr_name (string escape -- $abbr_name) - if string match -q "* *" -- $abbr_name - set -l msg ( _ "%s %s: Abbreviation %s cannot have spaces in the word\n" ) - printf $msg abbr --add $escaped_abbr_name >&2 - return 1 - end - - set -l abbr_val "$argv[2..-1]" - set -l abbr_var_name _fish_abbr_(string escape --style=var -- $abbr_name) - - if not set -q $abbr_var_name - # We default to the universal scope if the user didn't explicitly specify a scope and the - # abbreviation isn't already defined. - set -q abbr_scope[1] - or set abbr_scope --universal - end - true # make sure the next `set` command doesn't leak the previous status - set $abbr_scope $abbr_var_name $abbr_val -end - -function __fish_abbr_erase --no-scope-shadowing - if set -q argv[2] - printf ( _ "%s %s: Expected one argument\n" ) abbr --erase >&2 - return 1 - end - - # Because of the way abbreviations are expanded there can't be any spaces in the key. - set -l abbr_name $argv[1] - set -l escaped_name (string escape -- $abbr_name) - if string match -q "* *" -- $abbr_old_name - set -l msg ( _ "%s %s: Abbreviation %s cannot have spaces in the word\n" ) - printf $msg abbr --erase $escaped_name >&2 - return 1 - end - - set -l abbr_var_name _fish_abbr_(string escape --style=var -- $abbr_name) - - if not set -q $abbr_var_name - printf ( _ "%s %s: No abbreviation named %s\n" ) abbr --erase $escaped_name >&2 - return 121 - end - - set -e $abbr_var_name -end - -function __fish_abbr_rename --no-scope-shadowing - if test (count $argv) -ne 2 - printf ( _ "%s %s: Requires exactly two arguments\n" ) abbr --rename >&2 - return 1 - end - - set -l old_name $argv[1] - set -l new_name $argv[2] - set -l escaped_old_name (string escape -- $old_name) - set -l escaped_new_name (string escape -- $new_name) - if string match -q "* *" -- $old_name - set -l msg ( _ "%s %s: Abbreviation %s cannot have spaces in the word\n" ) - printf $msg abbr --rename $escaped_old_name >&2 - return 1 - end - if string match -q "* *" -- $new_name - set -l msg ( _ "%s %s: Abbreviation %s cannot have spaces in the word\n" ) - printf $msg abbr --rename $escaped_new_name >&2 - return 1 - end - - set -l old_var_name _fish_abbr_(string escape --style=var -- $old_name) - set -l new_var_name _fish_abbr_(string escape --style=var -- $new_name) - - if not set -q $old_var_name - printf ( _ "%s %s: No abbreviation named %s\n" ) abbr --rename $escaped_old_name >&2 - return 1 - end - if set -q $new_var_name - set -l msg ( _ "%s %s: Abbreviation %s already exists, cannot rename %s\n" ) - printf $msg abbr --rename $escaped_new_name $escaped_old_name >&2 - return 1 - end - - set -l old_var_val $$old_var_name - - if not set -q abbr_scope[1] - # User isn't forcing the scope so use the existing scope. - if set -ql $old_var_name - set abbr_scope --global - else - set abbr_scope --universal - end - end - - set -e $old_var_name - set $abbr_scope $new_var_name $old_var_val -end - -function __fish_abbr_list --no-scope-shadowing - if set -q argv[1] - printf ( _ "%s %s: Unexpected argument -- '%s'\n" ) abbr --erase $argv[1] >&2 - return 1 - end - - for var_name in (set --names) - string match -q '_fish_abbr_*' $var_name - or continue - - set -l abbr_name (string unescape --style=var (string sub -s 12 $var_name)) - echo $abbr_name - end -end - -function __fish_abbr_show --no-scope-shadowing - if set -q argv[1] - printf ( _ "%s %s: Unexpected argument -- '%s'\n" ) abbr --erase $argv[1] >&2 - return 1 - end - - for var_name in (set --names) - string match -q '_fish_abbr_*' $var_name - or continue - - set -l abbr_var_name $var_name - set -l abbr_name (string unescape --style=var -- (string sub -s 12 $abbr_var_name)) - set -l abbr_name (string escape --style=script -- $abbr_name) - set -l abbr_val $$abbr_var_name - set -l abbr_val (string escape --style=script -- $abbr_val) - - if set -ql $abbr_var_name - printf 'abbr -a %s -- %s %s\n' -l $abbr_name $abbr_val - end - if set -qg $abbr_var_name - printf 'abbr -a %s -- %s %s\n' -g $abbr_name $abbr_val - end - if set -qU $abbr_var_name - printf 'abbr -a %s -- %s %s\n' -U $abbr_name $abbr_val - end - end + return 1 end diff --git a/share/functions/abbr_old.fish b/share/functions/abbr_old.fish deleted file mode 100644 index c43f344ea..000000000 --- a/share/functions/abbr_old.fish +++ /dev/null @@ -1,206 +0,0 @@ -function abbr_old --description "Manage abbreviations using old fish 2.x scheme." - # parse arguments - set -l mode - set -l mode_flag # the flag that was specified, for better errors - set -l mode_arg - set -l needs_arg no - while set -q argv[1] - set -l new_mode - switch $argv[1] - case '-h' '--help' - __fish_print_help abbr - return 0 - case '-a' '--add' - set new_mode add - set needs_arg multi - case '-r' '--rename' - set new_mode rename - set needs_arg double - case '-e' '--erase' - set new_mode erase - set needs_arg single - case '-l' '--list' - set new_mode list - case '-s' '--show' - set new_mode show - case '--' - set -e argv[1] - break - case '-*' - printf ( _ "%s: invalid option -- %s\n" ) abbr $argv[1] >&2 - return 1 - case '*' - break - end - if test -n "$mode" -a -n "$new_mode" - # we're trying to set two different modes - printf ( _ "%s: %s cannot be specified along with %s\n" ) abbr $argv[1] $mode_flag >&2 - return 1 - end - set mode $new_mode - set mode_flag $argv[1] - set -e argv[1] - end - - # If run with no options, treat it like --add if we have an argument, or - # --show if we do not have an argument - if not set -q mode[1] - if set -q argv[1] - set mode add - set needs_arg multi - else - set mode show - end - end - - if test $needs_arg = single - set mode_arg $argv[1] - set needs_arg no - set -e argv[1] - else if test $needs_arg = double - # Pull the two parameters from argv. - # * leave argv non-empty, if there are more than two arguments - # * leave needs_arg set to double if there is not enough arguments - if set -q argv[1] - set param1 $argv[1] - set -e argv[1] - if set -q argv[1] - set param2 $argv[1] - set needs_arg no - set -e argv[1] - end - end - else if test $needs_arg = multi - set mode_arg $argv - set needs_arg no - set -e argv - end - if test $needs_arg != no - printf ( _ "%s: option requires an argument -- %s\n" ) abbr $mode_flag >&2 - return 1 - end - - # none of our modes want any excess arguments - if set -q argv[1] - printf ( _ "%s: Unexpected argument -- %s\n" ) abbr $argv[1] >&2 - return 1 - end - - switch $mode - case 'add' - # Convert from old "key=value" syntax - # TODO: This should be removed later - if not set -q mode_arg[2] - and string match -qr '^[^ ]+=' -- $mode_arg - set mode_arg (string split "=" -- $mode_arg) - end - - # Bail out early if the exact abbr is already in - set -q fish_user_abbreviations - and contains -- "$mode_arg" $fish_user_abbreviations - and return 0 - - set -l key $mode_arg[1] - set -e mode_arg[1] - set -l value "$mode_arg" - # Because we later store "$key $value", there can't be any spaces in the key - if string match -q "* *" -- $key - printf ( _ "%s: abbreviation cannot have spaces in the key\n" ) abbr >&2 - return 1 - end - if test -z "$value" - printf ( _ "%s: abbreviation must have a value\n" ) abbr >&2 - return 1 - end - if set -l idx (__fish_abbr_get_by_key $key) - # erase the existing abbreviation - set -e fish_user_abbreviations[$idx] - end - if not set -q fish_user_abbreviations - # initialize as a universal variable, so we can skip the -U later - # and therefore work properly if someone sets this as a global variable - set -U fish_user_abbreviations - end - set fish_user_abbreviations $fish_user_abbreviations "$key $value" - return 0 - - case 'rename' - set -l old_name $param1 - set -l new_name $param2 - - # if the target name already exists, throw an error - if set -l idx (__fish_abbr_get_by_key $new_name) - printf ( _ "%s: abbreviation '%s' already exists, cannot rename\n" ) abbr $new_name >&2 - return 2 - end - - # Because we later store "$key $value", there can't be any spaces in the key - if string match -q "* *" -- $new_name - printf ( _ "%s: abbreviation cannot have spaces in the key\n" ) abbr >&2 - return 1 - end - - set -l idx (__fish_abbr_get_by_key $old_name) - or begin - printf ( _ "%s: no such abbreviation '%s'\n" ) abbr $old_name >&2 - return 2 - end - - set -l value (string split " " -m 1 -- $fish_user_abbreviations[$idx])[2] - set fish_user_abbreviations[$idx] "$new_name $value" - return 0 - - case 'erase' - if set -l idx (__fish_abbr_get_by_key $mode_arg) - set -e fish_user_abbreviations[$idx] - return 0 - else - printf ( _ "%s: no such abbreviation '%s'\n" ) abbr $mode_arg >&2 - return 2 - end - - case 'show' - for i in $fish_user_abbreviations - set -l opt_double_dash - set -l kv (string split " " -m 1 -- $i) - set -l key $kv[1] - set -l value $kv[2] - - # Check to see if either key or value has a leading dash - # If so, we need to write -- - string match -q -- '-*' $key $value - and set opt_double_dash '--' - echo abbr $opt_double_dash (string escape -- $key $value) - end - return 0 - - case 'list' - for i in $fish_user_abbreviations - set -l key (string split " " -m 1 -- $i)[1] - printf "%s\n" $key - end - return 0 - end -end - -function __fish_abbr_get_by_key - if not set -q argv[1] - echo "__fish_abbr_get_by_key: expected one argument, got none" >&2 - return 2 - end - - set -q fish_user_abbreviations - or return 1 - - # Going through all entries is still quicker than calling `seq` - set -l keys - for kv in $fish_user_abbreviations - # If this does not match, we have screwed up before and the error should be reported - set keys $keys (string split " " -m 1 -- $kv)[1] - end - if set -l idx (contains -i -- $argv[1] $keys) - echo $idx - return 0 - end - return 1 -end diff --git a/src/env.cpp b/src/env.cpp index 0fd675539..78512c23e 100644 --- a/src/env.cpp +++ b/src/env.cpp @@ -41,7 +41,6 @@ #include "env.h" #include "env_universal_common.h" #include "event.h" -#include "expand.h" #include "fallback.h" // IWYU pragma: keep #include "fish_version.h" #include "history.h" @@ -188,7 +187,7 @@ void var_stack_t::push(bool new_scope) { // i.e. not if it's just `begin; end` or "--no-scope-shadowing". if (new_scope) { if (!(top_node == this->global_env)) { - for (auto &var : top_node->env) { + for (auto& var : top_node->env) { if (var.second.exportv) { // This should copy var node->env.insert(var); @@ -604,7 +603,7 @@ static bool variable_is_colon_delimited_var(const wcstring &str) { } /// React to modifying the given variable. -static void react_to_variable_change(const wchar_t *op, const wcstring &key) { +static void react_to_variable_change(const wcstring &key) { // Don't do any of this until `env_init()` has run. We only want to do this in response to // variables set by the user; e.g., in a script like *config.fish* or interactively or as part // of loading the universal variables for the first time. @@ -632,36 +631,37 @@ static void react_to_variable_change(const wchar_t *op, const wcstring &key) { env_set_read_limit(); } else if (key == L"FISH_HISTORY") { reader_change_history(history_session_id().c_str()); - } else if (wcsncmp(key.c_str(), L"_fish_abbr_", wcslen(L"_fish_abbr_")) == 0) { - update_abbr_cache(op, key); } } /// Universal variable callback function. This function makes sure the proper events are triggered /// when an event occurs. static void universal_callback(fish_message_type_t type, const wchar_t *name) { - const wchar_t *op; + const wchar_t *str = NULL; switch (type) { case SET: case SET_EXPORT: { - op = L"SET"; + str = L"SET"; break; } case ERASE: { - op = L"ERASE"; + str = L"ERASE"; break; } } - react_to_variable_change(op, name); - vars_stack().mark_changed_exported(); + if (str) { + vars_stack().mark_changed_exported(); - event_t ev = event_t::variable_event(name); - ev.arguments.push_back(L"VARIABLE"); - ev.arguments.push_back(op); - ev.arguments.push_back(name); - event_fire(&ev); + event_t ev = event_t::variable_event(name); + ev.arguments.push_back(L"VARIABLE"); + ev.arguments.push_back(str); + ev.arguments.push_back(name); + event_fire(&ev); + } + + if (name) react_to_variable_change(name); } /// Make sure the PATH variable contains something. @@ -1130,7 +1130,7 @@ int env_set(const wcstring &key, const wchar_t *val, env_mode_flags_t var_mode) event_fire(&ev); // debug( 1, L"env_set: return from event firing" ); - react_to_variable_change(L"SET", key); + react_to_variable_change(key); return ENV_OK; } @@ -1203,7 +1203,7 @@ int env_remove(const wcstring &key, int var_mode) { if (is_exported) vars_stack().mark_changed_exported(); } - react_to_variable_change(L"ERASE", key); + react_to_variable_change(key); return !erased; } diff --git a/src/expand.cpp b/src/expand.cpp index 347f0c3c3..e00a1bff0 100644 --- a/src/expand.cpp +++ b/src/expand.cpp @@ -1572,39 +1572,37 @@ bool fish_xdm_login_hack_hack_hack_hack(std::vector *cmds, int argc return result; } -std::map abbreviations; -void update_abbr_cache(const wchar_t *op, const wcstring varname) { - wcstring abbr; - if (!unescape_string(varname.substr(wcslen(L"_fish_abbr_")), &abbr, 0, STRING_STYLE_VAR)) { - debug(1, L"Abbreviation var '%ls' is not correctly encoded, ignoring it.", varname.c_str()); - return; - } - abbreviations.erase(abbr); - if (wcscmp(op, L"ERASE") != 0) { - const env_var_t expansion = env_get_string(varname); - if (!expansion.missing_or_empty()) { - abbreviations.emplace(std::make_pair(abbr, expansion)); - } - } -} - bool expand_abbreviation(const wcstring &src, wcstring *output) { if (src.empty()) return false; - auto abbr = abbreviations.find(src); - if (abbr == abbreviations.end()) return false; - if (output != NULL) output->assign(abbr->second); - return true; + // Get the abbreviations. Return false if we have none. + env_var_t abbrs = env_get_string(USER_ABBREVIATIONS_VARIABLE_NAME); + if (abbrs.missing_or_empty()) return false; -#if 0 - for (auto abbr : abbreviations) { - if (src == abbr.first) { - // We found a matching abbreviation. Set output to the expansion. - if (output != NULL) output->assign(abbr.second); - return true; + bool result = false; + std::vector abbrsv; + tokenize_variable_array(abbrs, abbrsv); + for (auto abbr : abbrsv) { + // Abbreviation is expected to be of the form 'foo=bar' or 'foo bar'. Parse out the first = + // or space. Silently skip on failure (no equals, or equals at the end or beginning). Try to + // avoid copying any strings until we are sure this is a match. + size_t equals_pos = abbr.find(L'='); + size_t space_pos = abbr.find(L' '); + size_t separator = mini(equals_pos, space_pos); + if (separator == wcstring::npos || separator == 0 || separator + 1 == abbr.size()) continue; + + // Find the character just past the end of the command. Walk backwards, skipping spaces. + size_t cmd_end = separator; + while (cmd_end > 0 && iswspace(abbr.at(cmd_end - 1))) cmd_end--; + + // See if this command matches. + if (abbr.compare(0, cmd_end, src) == 0) { + // Success. Set output to everything past the end of the string. + if (output != NULL) output->assign(abbr, separator + 1, wcstring::npos); + + result = true; + break; } } - - return false; -#endif + return result; } diff --git a/src/expand.h b/src/expand.h index 10fc88df5..a904090e7 100644 --- a/src/expand.h +++ b/src/expand.h @@ -135,10 +135,11 @@ wcstring replace_home_directory_with_tilde(const wcstring &str); /// Abbreviation support. Expand src as an abbreviation, returning true if one was found, false if /// not. If result is not-null, returns the abbreviation by reference. -void update_abbr_cache(const wchar_t *op, const wcstring varnam); +#define USER_ABBREVIATIONS_VARIABLE_NAME L"fish_user_abbreviations" bool expand_abbreviation(const wcstring &src, wcstring *output); // Terrible hacks bool fish_xdm_login_hack_hack_hack_hack(std::vector *cmds, int argc, const char *const *argv); + #endif diff --git a/src/fish_tests.cpp b/src/fish_tests.cpp index 0702efb9c..49390d2ba 100644 --- a/src/fish_tests.cpp +++ b/src/fish_tests.cpp @@ -1610,15 +1610,21 @@ static void test_fuzzy_match(void) { static void test_abbreviations(void) { say(L"Testing abbreviations"); + + const wchar_t *abbreviations = + L"gc=git checkout" ARRAY_SEP_STR + L"foo=" ARRAY_SEP_STR + L"gc=something else" ARRAY_SEP_STR + L"=" ARRAY_SEP_STR + L"=foo" ARRAY_SEP_STR + L"foo" ARRAY_SEP_STR + L"foo=bar" ARRAY_SEP_STR + L"gx git checkout"; + env_push(true); - const std::vector> abbreviations = { - {L"gc", L"git checkout"}, {L"foo", L"bar"}, {L"gx", L"git checkout"}, - }; - for (auto it : abbreviations) { - int ret = env_set(L"_fish_abbr_" + it.first, it.second.c_str(), ENV_LOCAL); - if (ret != 0) err(L"Unable to set abbreviation variable"); - } + int ret = env_set(USER_ABBREVIATIONS_VARIABLE_NAME, abbreviations, ENV_LOCAL); + if (ret != 0) err(L"Unable to set abbreviation variable"); wcstring result; if (expand_abbreviation(L"", &result)) err(L"Unexpected success with empty abbreviation"); diff --git a/tests/abbr.err b/tests/abbr.err index 9b8a4b9d1..83f97ab79 100644 --- a/tests/abbr.err +++ b/tests/abbr.err @@ -1,52 +1,7 @@ - -#################### -# Test basic add and list of __abbr1 - -#################### -# Erasing one that doesn't exist should do nothing -abbr --erase: No abbreviation named NOT_AN_ABBR - -#################### -# Adding existing __abbr1 should be idempotent - -#################### -# Replacing __abbr1 definition - -#################### -# __abbr1 -s and --show tests - -#################### -# Test erasing __abbr1 - -#################### -# Ensure we escape special characters on output - -#################### -# Ensure we handle leading dashes in abbreviation names properly - -#################### -# Test that an abbr word containing spaces is rejected -abbr --add: Abbreviation 'a b c' cannot have spaces in the word - -#################### -# Test renaming - -#################### -# Test renaming a nonexistent abbreviation -abbr --rename: No abbreviation named __abbr6 - -#################### -# Test renaming to a abbreviation with spaces -abbr --rename: Abbreviation 'g h i' cannot have spaces in the word - -#################### -# Test renaming without arguments -abbr --rename: Requires exactly two arguments - -#################### -# Test renaming with too many arguments -abbr --rename: Requires exactly two arguments - -#################### -# Test renaming to existing abbreviation -abbr --rename: Abbreviation __abbr12 already exists, cannot rename __abbr11 +abbr: no such abbreviation 'NOT_AN_ABBR' +abbr: abbreviation cannot have spaces in the key +abbr: no such abbreviation '__abbr6' +abbr: abbreviation cannot have spaces in the key +abbr: option requires an argument -- -r +abbr: Unexpected argument -- __abbr10 +abbr: abbreviation '__abbr12' already exists, cannot rename diff --git a/tests/abbr.in b/tests/abbr.in index 42fb83c94..26b421171 100644 --- a/tests/abbr.in +++ b/tests/abbr.in @@ -1,42 +1,44 @@ -logmsg Test basic add and list of __abbr1 +# Test basic add and list abbr __abbr1 alpha beta gamma abbr | grep __abbr1 -logmsg Erasing one that doesn\'t exist should do nothing +# Erasing one that doesn't exist should do nothing abbr --erase NOT_AN_ABBR abbr | grep __abbr1 -logmsg Adding existing __abbr1 should be idempotent +# Adding existing one should be idempotent abbr __abbr1 alpha beta gamma abbr | grep __abbr1 -logmsg Replacing __abbr1 definition +# Replacing abbr __abbr1 delta abbr | grep __abbr1 -logmsg __abbr1 -s and --show tests +# -s and --show tests abbr -s | grep __abbr1 abbr --show | grep __abbr1 -logmsg Test erasing __abbr1 +# Test erasing abbr -e __abbr1 abbr | grep __abbr1 -logmsg Ensure we escape special characters on output +# Ensure we escape special characters on output abbr '~__abbr2' '$xyz' abbr | grep __abbr2 abbr -e '~__abbr2' -logmsg Ensure we handle leading dashes in abbreviation names properly +# Ensure we handle leading dashes in abbreviation names properly abbr -- '--__abbr3' 'xyz' abbr | grep __abbr3 abbr -e -- '--__abbr3' -logmsg Test that an abbr word containing spaces is rejected -abbr "a b c" "d e f" -abbr | grep 'a b c' +# Ensure we are not recognizing later "=" as separators +abbr d2 env a=b banana +abbr -l | string match -q d2; or echo "= test failed" -logmsg Test renaming +abbr "a b c" "d e f"; or true + +# Test renaming abbr __abbr4 omega abbr | grep __abbr5 abbr -r __abbr4 __abbr5 @@ -44,28 +46,26 @@ abbr | grep __abbr5 abbr -e __abbr5 abbr | grep __abbr4 -logmsg Test renaming a nonexistent abbreviation -abbr -r __abbr6 __abbr +# Test renaming a nonexistent abbreviation +abbr -r __abbr6 __abbr; or true -logmsg Test renaming to a abbreviation with spaces +# Test renaming to a abbreviation with spaces abbr __abbr4 omega -abbr -r __abbr4 "g h i" +abbr -r __abbr4 "g h i"; or true abbr -e __abbr4 -logmsg Test renaming without arguments +# Test renaming without arguments abbr __abbr7 omega -abbr -r __abbr7 +abbr -r __abbr7; or true -logmsg Test renaming with too many arguments +# Test renaming with too many arguments abbr __abbr8 omega -abbr -r __abbr8 __abbr9 __abbr10 +abbr -r __abbr8 __abbr9 __abbr10; or true abbr | grep __abbr8 -abbr | grep __abbr9 -abbr | grep __abbr10 +abbr | grep __abbr9; or true +abbr | grep __abbr10; or true -logmsg Test renaming to existing abbreviation +# Test renaming to existing abbreviation abbr __abbr11 omega11 abbr __abbr12 omega12 -abbr -r __abbr11 __abbr12 - -exit 0 # the last `abbr` command is expected to fail -- don't let that cause a test failure +abbr -r __abbr11 __abbr12; or true diff --git a/tests/abbr.out b/tests/abbr.out index c468b15f6..168fbfcee 100644 --- a/tests/abbr.out +++ b/tests/abbr.out @@ -1,55 +1,10 @@ - -#################### -# Test basic add and list of __abbr1 -abbr -a -U -- __abbr1 'alpha beta gamma' - -#################### -# Erasing one that doesn't exist should do nothing -abbr -a -U -- __abbr1 'alpha beta gamma' - -#################### -# Adding existing __abbr1 should be idempotent -abbr -a -U -- __abbr1 'alpha beta gamma' - -#################### -# Replacing __abbr1 definition -abbr -a -U -- __abbr1 delta - -#################### -# __abbr1 -s and --show tests -abbr -a -U -- __abbr1 delta -abbr -a -U -- __abbr1 delta - -#################### -# Test erasing __abbr1 - -#################### -# Ensure we escape special characters on output -abbr -a -U -- '~__abbr2' '$xyz' - -#################### -# Ensure we handle leading dashes in abbreviation names properly -abbr -a -U -- --__abbr3 xyz - -#################### -# Test that an abbr word containing spaces is rejected - -#################### -# Test renaming -abbr -a -U -- __abbr5 omega - -#################### -# Test renaming a nonexistent abbreviation - -#################### -# Test renaming to a abbreviation with spaces - -#################### -# Test renaming without arguments - -#################### -# Test renaming with too many arguments -abbr -a -U -- __abbr8 omega - -#################### -# Test renaming to existing abbreviation +abbr __abbr1 'alpha beta gamma' +abbr __abbr1 'alpha beta gamma' +abbr __abbr1 'alpha beta gamma' +abbr __abbr1 delta +abbr __abbr1 delta +abbr __abbr1 delta +abbr '~__abbr2' '$xyz' +abbr -- --__abbr3 xyz +abbr __abbr5 omega +abbr __abbr8 omega diff --git a/tests/functions.in b/tests/functions.in index d1eae6e64..4e2f03433 100644 --- a/tests/functions.in +++ b/tests/functions.in @@ -46,7 +46,7 @@ or not string match -q '*/share/functions/abbr.fish' $x[1] or test $x[2] != autoloaded or test $x[3] != 1 or test $x[4] != scope-shadowing -or test $x[5] != 'Manage abbreviations using new fish 3.0 scheme.' +or test $x[5] != 'Manage abbreviations' echo "Unexpected output for 'functions -v -D abbr': $x" >&2 end From 886fc38d55b04bb43e69860004d22bd246825421 Mon Sep 17 00:00:00 2001 From: Kurtis Rader Date: Sun, 13 Aug 2017 16:15:12 -0700 Subject: [PATCH 79/79] fix typo introduced by an earlier revert --- src/builtin_set.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/builtin_set.cpp b/src/builtin_set.cpp index a5f3aada7..81b66d65b 100644 --- a/src/builtin_set.cpp +++ b/src/builtin_set.cpp @@ -203,7 +203,7 @@ static int validate_cmd_opts(const wchar_t *cmd, set_cmd_opts_t &opts, //!OCLIN static int check_global_scope_exists(const wchar_t *cmd, set_cmd_opts_t &opts, const wchar_t *dest, io_streams_t &streams) { if (opts.universal) { - env_var_t global_dest = env_get(dest, ENV_GLOBAL); + env_var_t global_dest = env_get_string(dest, ENV_GLOBAL); if (!global_dest.missing() && shell_is_interactive()) { streams.err.append_format(BUILTIN_SET_UVAR_ERR, cmd, dest); }