| Message ID | 20210719123732.24457-1-alex.bennee@linaro.org |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [RFC] tcg/plugins: implement a qemu_plugin_user_exit helper | expand |
Alex Bennée <alex.bennee@linaro.org> writes: > In user-mode emulation there is a small race between preexit_cleanup > and exit_group() which means we may end up calling instrumented > instructions before the kernel reaps child threads. To solve this we > implement a new helper which ensures the callbacks are flushed along > with any translations before we let the host do it's a thing. > > While we are at it make the documentation of > qemu_plugin_register_atexit_cb clearer as to what the user can expect. > <snip> > > +/* > + * Handle exit from linux-user. Unlike the normal atexit() mechanism > + * we need to handle the clean-up manually as it's possible threads > + * are still running. We need to remove all callbacks from code > + * generation, flush the current translations and then we can safely > + * trigger the exit callbacks. > + */ > + > +void qemu_plugin_user_exit(void) > +{ > + enum qemu_plugin_event ev; > + > + QEMU_LOCK_GUARD(&plugin.lock); > + > + start_exclusive(); > + > + /* un-register all callbacks except the final AT_EXIT one */ > + for (ev = 0; ev < QEMU_PLUGIN_EV_MAX; ev++) { > + if (ev != QEMU_PLUGIN_EV_ATEXIT) { > + struct qemu_plugin_ctx *ctx; > + QTAILQ_FOREACH(ctx, &plugin.ctxs, entry) { > + plugin_unregister_cb__locked(ctx, ev); > + } > + } > + } > + > + tb_flush(current_cpu); We also need to disable memory helpers during the exclusive period as that is another route into a callback: --8<---------------cut here---------------start------------->8--- modified plugins/core.c @@ -498,6 +499,7 @@ void qemu_plugin_register_atexit_cb(qemu_plugin_id_t id, void qemu_plugin_user_exit(void) { enum qemu_plugin_event ev; + CPUState *cpu; QEMU_LOCK_GUARD(&plugin.lock); @@ -514,6 +516,11 @@ void qemu_plugin_user_exit(void) } tb_flush(current_cpu); + + CPU_FOREACH(cpu) { + qemu_plugin_disable_mem_helpers(cpu); + } + end_exclusive(); /* now it's safe to handle the exit case */ --8<---------------cut here---------------end--------------->8--- > + end_exclusive(); > + > + /* now it's safe to handle the exit case */ > + qemu_plugin_atexit_cb(); > +} > + > /* > * Call this function after longjmp'ing to the main loop. It's possible that the > * last instruction of a TB might have used helpers, and therefore the -- Alex Bennée
On Mon, Jul 19, 2021, 7:57 AM Alex Bennée <alex.bennee@linaro.org> wrote: > > Alex Bennée <alex.bennee@linaro.org> writes: > > > In user-mode emulation there is a small race between preexit_cleanup > > and exit_group() which means we may end up calling instrumented > > instructions before the kernel reaps child threads. To solve this we > > implement a new helper which ensures the callbacks are flushed along > > with any translations before we let the host do it's a thing. > > > > While we are at it make the documentation of > > qemu_plugin_register_atexit_cb clearer as to what the user can expect. > > > <snip> > > > > +/* > > + * Handle exit from linux-user. Unlike the normal atexit() mechanism > > + * we need to handle the clean-up manually as it's possible threads > > + * are still running. We need to remove all callbacks from code > > + * generation, flush the current translations and then we can safely > > + * trigger the exit callbacks. > > + */ > > + > > +void qemu_plugin_user_exit(void) > > +{ > > + enum qemu_plugin_event ev; > > + > > + QEMU_LOCK_GUARD(&plugin.lock); > > + > > + start_exclusive(); > > + > > + /* un-register all callbacks except the final AT_EXIT one */ > > + for (ev = 0; ev < QEMU_PLUGIN_EV_MAX; ev++) { > > + if (ev != QEMU_PLUGIN_EV_ATEXIT) { > > + struct qemu_plugin_ctx *ctx; > > + QTAILQ_FOREACH(ctx, &plugin.ctxs, entry) { > > + plugin_unregister_cb__locked(ctx, ev); > > + } > > + } > > + } > > + > > + tb_flush(current_cpu); > > We also need to disable memory helpers during the exclusive period as > that is another route into a callback: > --8<---------------cut here---------------start------------->8--- > modified plugins/core.c > @@ -498,6 +499,7 @@ void qemu_plugin_register_atexit_cb(qemu_plugin_id_t > id, > void qemu_plugin_user_exit(void) > { > enum qemu_plugin_event ev; > + CPUState *cpu; > > QEMU_LOCK_GUARD(&plugin.lock); > > @@ -514,6 +516,11 @@ void qemu_plugin_user_exit(void) > } > > tb_flush(current_cpu); > + > + CPU_FOREACH(cpu) { > + qemu_plugin_disable_mem_helpers(cpu); > + } > + > end_exclusive(); > > /* now it's safe to handle the exit case */ > --8<---------------cut here---------------end--------------->8--- > I think both of these are find from a bsd-user point of view. Warner > > + end_exclusive(); > > + > > + /* now it's safe to handle the exit case */ > > + qemu_plugin_atexit_cb(); > > +} > > + > > /* > > * Call this function after longjmp'ing to the main loop. It's possible > that the > > * last instruction of a TB might have used helpers, and therefore the > > > -- > Alex Bennée > <div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 19, 2021, 7:57 AM Alex Bennée <<a href="mailto:alex.bennee@linaro.org">alex.bennee@linaro.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br> Alex Bennée <<a href="mailto:alex.bennee@linaro.org" target="_blank" rel="noreferrer">alex.bennee@linaro.org</a>> writes:<br> <br> > In user-mode emulation there is a small race between preexit_cleanup<br> > and exit_group() which means we may end up calling instrumented<br> > instructions before the kernel reaps child threads. To solve this we<br> > implement a new helper which ensures the callbacks are flushed along<br> > with any translations before we let the host do it's a thing.<br> ><br> > While we are at it make the documentation of<br> > qemu_plugin_register_atexit_cb clearer as to what the user can expect.<br> ><br> <snip><br> > <br> > +/*<br> > + * Handle exit from linux-user. Unlike the normal atexit() mechanism<br> > + * we need to handle the clean-up manually as it's possible threads<br> > + * are still running. We need to remove all callbacks from code<br> > + * generation, flush the current translations and then we can safely<br> > + * trigger the exit callbacks.<br> > + */<br> > +<br> > +void qemu_plugin_user_exit(void)<br> > +{<br> > + enum qemu_plugin_event ev;<br> > +<br> > + QEMU_LOCK_GUARD(&plugin.lock);<br> > +<br> > + start_exclusive();<br> > +<br> > + /* un-register all callbacks except the final AT_EXIT one */<br> > + for (ev = 0; ev < QEMU_PLUGIN_EV_MAX; ev++) {<br> > + if (ev != QEMU_PLUGIN_EV_ATEXIT) {<br> > + struct qemu_plugin_ctx *ctx;<br> > + QTAILQ_FOREACH(ctx, &plugin.ctxs, entry) {<br> > + plugin_unregister_cb__locked(ctx, ev);<br> > + }<br> > + }<br> > + }<br> > +<br> > + tb_flush(current_cpu);<br> <br> We also need to disable memory helpers during the exclusive period as<br> that is another route into a callback:<br> --8<---------------cut here---------------start------------->8---<br> modified plugins/core.c<br> @@ -498,6 +499,7 @@ void qemu_plugin_register_atexit_cb(qemu_plugin_id_t id,<br> void qemu_plugin_user_exit(void)<br> {<br> enum qemu_plugin_event ev;<br> + CPUState *cpu;<br> <br> QEMU_LOCK_GUARD(&plugin.lock);<br> <br> @@ -514,6 +516,11 @@ void qemu_plugin_user_exit(void)<br> }<br> <br> tb_flush(current_cpu);<br> +<br> + CPU_FOREACH(cpu) {<br> + qemu_plugin_disable_mem_helpers(cpu);<br> + }<br> +<br> end_exclusive();<br> <br> /* now it's safe to handle the exit case */<br> --8<---------------cut here---------------end--------------->8---<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">I think both of these are find from a bsd-user point of view.</div><div dir="auto"><br></div><div dir="auto">Warner </div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br> > + end_exclusive();<br> > +<br> > + /* now it's safe to handle the exit case */<br> > + qemu_plugin_atexit_cb();<br> > +}<br> > +<br> > /*<br> > * Call this function after longjmp'ing to the main loop. It's possible that the<br> > * last instruction of a TB might have used helpers, and therefore the<br> <br> <br> -- <br> Alex Bennée<br> </blockquote></div></div></div>
Warner Losh <imp@bsdimp.com> writes: > On Mon, Jul 19, 2021, 7:57 AM Alex Bennée <alex.bennee@linaro.org> wrote: > > Alex Bennée <alex.bennee@linaro.org> writes: > > > In user-mode emulation there is a small race between preexit_cleanup > > and exit_group() which means we may end up calling instrumented > > instructions before the kernel reaps child threads. To solve this we > > implement a new helper which ensures the callbacks are flushed along > > with any translations before we let the host do it's a thing. > > > > While we are at it make the documentation of > > qemu_plugin_register_atexit_cb clearer as to what the user can expect. > > > <snip> > > > > +/* > > + * Handle exit from linux-user. Unlike the normal atexit() mechanism > > + * we need to handle the clean-up manually as it's possible threads > > + * are still running. We need to remove all callbacks from code > > + * generation, flush the current translations and then we can safely > > + * trigger the exit callbacks. > > + */ > > + > > +void qemu_plugin_user_exit(void) > > +{ > > + enum qemu_plugin_event ev; > > + > > + QEMU_LOCK_GUARD(&plugin.lock); > > + > > + start_exclusive(); > > + > > + /* un-register all callbacks except the final AT_EXIT one */ > > + for (ev = 0; ev < QEMU_PLUGIN_EV_MAX; ev++) { > > + if (ev != QEMU_PLUGIN_EV_ATEXIT) { > > + struct qemu_plugin_ctx *ctx; > > + QTAILQ_FOREACH(ctx, &plugin.ctxs, entry) { > > + plugin_unregister_cb__locked(ctx, ev); > > + } > > + } > > + } > > + > > + tb_flush(current_cpu); > > We also need to disable memory helpers during the exclusive period as > that is another route into a callback: > --8<---------------cut here---------------start------------->8--- > modified plugins/core.c > @@ -498,6 +499,7 @@ void qemu_plugin_register_atexit_cb(qemu_plugin_id_t id, > void qemu_plugin_user_exit(void) > { > enum qemu_plugin_event ev; > + CPUState *cpu; > > QEMU_LOCK_GUARD(&plugin.lock); > > @@ -514,6 +516,11 @@ void qemu_plugin_user_exit(void) > } > > tb_flush(current_cpu); > + > + CPU_FOREACH(cpu) { > + qemu_plugin_disable_mem_helpers(cpu); > + } > + > end_exclusive(); > > /* now it's safe to handle the exit case */ > --8<---------------cut here---------------end--------------->8--- > > I think both of these are find from a bsd-user point of view. Acked-by: or Reviewed-by:? -- Alex Bennée
On Mon, Jul 19, 2021, 1:22 PM Alex Bennée <alex.bennee@linaro.org> wrote: > > Warner Losh <imp@bsdimp.com> writes: > > > On Mon, Jul 19, 2021, 7:57 AM Alex Bennée <alex.bennee@linaro.org> > wrote: > > > > Alex Bennée <alex.bennee@linaro.org> writes: > > > > > In user-mode emulation there is a small race between preexit_cleanup > > > and exit_group() which means we may end up calling instrumented > > > instructions before the kernel reaps child threads. To solve this we > > > implement a new helper which ensures the callbacks are flushed along > > > with any translations before we let the host do it's a thing. > > > > > > While we are at it make the documentation of > > > qemu_plugin_register_atexit_cb clearer as to what the user can expect. > > > > > <snip> > > > > > > +/* > > > + * Handle exit from linux-user. Unlike the normal atexit() mechanism > > > + * we need to handle the clean-up manually as it's possible threads > > > + * are still running. We need to remove all callbacks from code > > > + * generation, flush the current translations and then we can safely > > > + * trigger the exit callbacks. > > > + */ > > > + > > > +void qemu_plugin_user_exit(void) > > > +{ > > > + enum qemu_plugin_event ev; > > > + > > > + QEMU_LOCK_GUARD(&plugin.lock); > > > + > > > + start_exclusive(); > > > + > > > + /* un-register all callbacks except the final AT_EXIT one */ > > > + for (ev = 0; ev < QEMU_PLUGIN_EV_MAX; ev++) { > > > + if (ev != QEMU_PLUGIN_EV_ATEXIT) { > > > + struct qemu_plugin_ctx *ctx; > > > + QTAILQ_FOREACH(ctx, &plugin.ctxs, entry) { > > > + plugin_unregister_cb__locked(ctx, ev); > > > + } > > > + } > > > + } > > > + > > > + tb_flush(current_cpu); > > > > We also need to disable memory helpers during the exclusive period as > > that is another route into a callback: > > --8<---------------cut here---------------start------------->8--- > > modified plugins/core.c > > @@ -498,6 +499,7 @@ void > qemu_plugin_register_atexit_cb(qemu_plugin_id_t id, > > void qemu_plugin_user_exit(void) > > { > > enum qemu_plugin_event ev; > > + CPUState *cpu; > > > > QEMU_LOCK_GUARD(&plugin.lock); > > > > @@ -514,6 +516,11 @@ void qemu_plugin_user_exit(void) > > } > > > > tb_flush(current_cpu); > > + > > + CPU_FOREACH(cpu) { > > + qemu_plugin_disable_mem_helpers(cpu); > > + } > > + > > end_exclusive(); > > > > /* now it's safe to handle the exit case */ > > --8<---------------cut here---------------end--------------->8--- > > > > I think both of these are find from a bsd-user point of view. > > Acked-by: or Reviewed-by:? > Sorry I wasn't clear. Acked-by: Warner Losh <imp@bsdimp.com> > -- > Alex Bennée > <div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 19, 2021, 1:22 PM Alex Bennée <<a href="mailto:alex.bennee@linaro.org">alex.bennee@linaro.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br> Warner Losh <<a href="mailto:imp@bsdimp.com" target="_blank" rel="noreferrer">imp@bsdimp.com</a>> writes:<br> <br> > On Mon, Jul 19, 2021, 7:57 AM Alex Bennée <<a href="mailto:alex.bennee@linaro.org" target="_blank" rel="noreferrer">alex.bennee@linaro.org</a>> wrote:<br> ><br> > Alex Bennée <<a href="mailto:alex.bennee@linaro.org" target="_blank" rel="noreferrer">alex.bennee@linaro.org</a>> writes:<br> ><br> > > In user-mode emulation there is a small race between preexit_cleanup<br> > > and exit_group() which means we may end up calling instrumented<br> > > instructions before the kernel reaps child threads. To solve this we<br> > > implement a new helper which ensures the callbacks are flushed along<br> > > with any translations before we let the host do it's a thing.<br> > ><br> > > While we are at it make the documentation of<br> > > qemu_plugin_register_atexit_cb clearer as to what the user can expect.<br> > ><br> > <snip><br> > > <br> > > +/*<br> > > + * Handle exit from linux-user. Unlike the normal atexit() mechanism<br> > > + * we need to handle the clean-up manually as it's possible threads<br> > > + * are still running. We need to remove all callbacks from code<br> > > + * generation, flush the current translations and then we can safely<br> > > + * trigger the exit callbacks.<br> > > + */<br> > > +<br> > > +void qemu_plugin_user_exit(void)<br> > > +{<br> > > + enum qemu_plugin_event ev;<br> > > +<br> > > + QEMU_LOCK_GUARD(&plugin.lock);<br> > > +<br> > > + start_exclusive();<br> > > +<br> > > + /* un-register all callbacks except the final AT_EXIT one */<br> > > + for (ev = 0; ev < QEMU_PLUGIN_EV_MAX; ev++) {<br> > > + if (ev != QEMU_PLUGIN_EV_ATEXIT) {<br> > > + struct qemu_plugin_ctx *ctx;<br> > > + QTAILQ_FOREACH(ctx, &plugin.ctxs, entry) {<br> > > + plugin_unregister_cb__locked(ctx, ev);<br> > > + }<br> > > + }<br> > > + }<br> > > +<br> > > + tb_flush(current_cpu);<br> ><br> > We also need to disable memory helpers during the exclusive period as<br> > that is another route into a callback:<br> > --8<---------------cut here---------------start------------->8---<br> > modified plugins/core.c<br> > @@ -498,6 +499,7 @@ void qemu_plugin_register_atexit_cb(qemu_plugin_id_t id,<br> > void qemu_plugin_user_exit(void)<br> > {<br> > enum qemu_plugin_event ev;<br> > + CPUState *cpu;<br> ><br> > QEMU_LOCK_GUARD(&plugin.lock);<br> ><br> > @@ -514,6 +516,11 @@ void qemu_plugin_user_exit(void)<br> > }<br> ><br> > tb_flush(current_cpu);<br> > +<br> > + CPU_FOREACH(cpu) {<br> > + qemu_plugin_disable_mem_helpers(cpu);<br> > + }<br> > +<br> > end_exclusive();<br> ><br> > /* now it's safe to handle the exit case */<br> > --8<---------------cut here---------------end--------------->8---<br> ><br> > I think both of these are find from a bsd-user point of view.<br> <br> Acked-by: or Reviewed-by:?<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Sorry I wasn't clear.</div><div dir="auto"><br></div><div dir="auto">Acked-by: Warner Losh <<a href="mailto:imp@bsdimp.com">imp@bsdimp.com</a>></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br> -- <br> Alex Bennée<br> </blockquote></div></div></div>
On Mon, Jul 19, 2021, 14:37 Alex Bennée <alex.bennee@linaro.org> wrote: > In user-mode emulation there is a small race between preexit_cleanup > and exit_group() which means we may end up calling instrumented > instructions before the kernel reaps child threads. To solve this we > implement a new helper which ensures the callbacks are flushed along > with any translations before we let the host do it's a thing. > > While we are at it make the documentation of > qemu_plugin_register_atexit_cb clearer as to what the user can expect. > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > Looks great to me and I cannot reproduce the race with it installed in. With calling `qemu_plugin_disable_mem_helpers`: Reviewed-by: Mahmoud Mandour <ma.mandourr@gmail.com> --- > include/qemu/plugin.h | 12 ++++++++++++ > include/qemu/qemu-plugin.h | 13 +++++++++++++ > bsd-user/syscall.c | 6 +++--- > linux-user/exit.c | 2 +- > plugins/core.c | 33 +++++++++++++++++++++++++++++++++ > 5 files changed, 62 insertions(+), 4 deletions(-) > > diff --git a/include/qemu/plugin.h b/include/qemu/plugin.h > index 0fefbc6084..9a8438f683 100644 > --- a/include/qemu/plugin.h > +++ b/include/qemu/plugin.h > @@ -190,6 +190,16 @@ void qemu_plugin_add_dyn_cb_arr(GArray *arr); > > void qemu_plugin_disable_mem_helpers(CPUState *cpu); > > +/** > + * qemu_plugin_user_exit(): clean-up callbacks before calling exit > callbacks > + * > + * This is a user-mode only helper that ensure we have fully cleared > + * callbacks from all threads before calling the exit callbacks. This > + * is so the plugins themselves don't have to jump through hoops to > + * guard against race conditions. > + */ > +void qemu_plugin_user_exit(void); > + > #else /* !CONFIG_PLUGIN */ > > static inline void qemu_plugin_add_opts(void) > @@ -250,6 +260,8 @@ void qemu_plugin_add_dyn_cb_arr(GArray *arr) > static inline void qemu_plugin_disable_mem_helpers(CPUState *cpu) > { } > > +static inline void qemu_plugin_user_exit(void) > +{ } > #endif /* !CONFIG_PLUGIN */ > > #endif /* QEMU_PLUGIN_H */ > diff --git a/include/qemu/qemu-plugin.h b/include/qemu/qemu-plugin.h > index dc3496f36c..e6e815abc5 100644 > --- a/include/qemu/qemu-plugin.h > +++ b/include/qemu/qemu-plugin.h > @@ -549,6 +549,19 @@ void qemu_plugin_vcpu_for_each(qemu_plugin_id_t id, > void qemu_plugin_register_flush_cb(qemu_plugin_id_t id, > qemu_plugin_simple_cb_t cb); > > +/** > + * qemu_plugin_register_atexit_cb() - register exit callback > + * @id: plugin ID > + * @cb: callback > + * @userdata: user data for callback > + * > + * The @cb function is called once execution has finished. Plugins > + * should be able to free all their resources at this point much like > + * after a reset/uninstall callback is called. > + * > + * In user-mode it is possible a few un-instrumented instructions from > + * child threads may run before the host kernel reaps the threads. > + */ > void qemu_plugin_register_atexit_cb(qemu_plugin_id_t id, > qemu_plugin_udata_cb_t cb, void > *userdata); > > diff --git a/bsd-user/syscall.c b/bsd-user/syscall.c > index 7d986e9700..3f44311396 100644 > --- a/bsd-user/syscall.c > +++ b/bsd-user/syscall.c > @@ -335,7 +335,7 @@ abi_long do_freebsd_syscall(void *cpu_env, int num, > abi_long arg1, > _mcleanup(); > #endif > gdb_exit(arg1); > - qemu_plugin_atexit_cb(); > + qemu_plugin_user_exit(); > /* XXX: should free thread stack and CPU env */ > _exit(arg1); > ret = 0; /* avoid warning */ > @@ -437,7 +437,7 @@ abi_long do_netbsd_syscall(void *cpu_env, int num, > abi_long arg1, > _mcleanup(); > #endif > gdb_exit(arg1); > - qemu_plugin_atexit_cb(); > + qemu_plugin_user_exit(); > /* XXX: should free thread stack and CPU env */ > _exit(arg1); > ret = 0; /* avoid warning */ > @@ -516,7 +516,7 @@ abi_long do_openbsd_syscall(void *cpu_env, int num, > abi_long arg1, > _mcleanup(); > #endif > gdb_exit(arg1); > - qemu_plugin_atexit_cb(); > + qemu_plugin_user_exit(); > /* XXX: should free thread stack and CPU env */ > _exit(arg1); > ret = 0; /* avoid warning */ > diff --git a/linux-user/exit.c b/linux-user/exit.c > index 70b344048c..527e29cbc1 100644 > --- a/linux-user/exit.c > +++ b/linux-user/exit.c > @@ -35,5 +35,5 @@ void preexit_cleanup(CPUArchState *env, int code) > __gcov_dump(); > #endif > gdb_exit(code); > - qemu_plugin_atexit_cb(); > + qemu_plugin_user_exit(); > } > diff --git a/plugins/core.c b/plugins/core.c > index e1bcdb570d..c573b81a96 100644 > --- a/plugins/core.c > +++ b/plugins/core.c > @@ -487,6 +487,39 @@ void qemu_plugin_register_atexit_cb(qemu_plugin_id_t > id, > plugin_register_cb_udata(id, QEMU_PLUGIN_EV_ATEXIT, cb, udata); > } > > +/* > + * Handle exit from linux-user. Unlike the normal atexit() mechanism > + * we need to handle the clean-up manually as it's possible threads > + * are still running. We need to remove all callbacks from code > + * generation, flush the current translations and then we can safely > + * trigger the exit callbacks. > + */ > + > +void qemu_plugin_user_exit(void) > +{ > + enum qemu_plugin_event ev; > + > + QEMU_LOCK_GUARD(&plugin.lock); > + > + start_exclusive(); > + > + /* un-register all callbacks except the final AT_EXIT one */ > + for (ev = 0; ev < QEMU_PLUGIN_EV_MAX; ev++) { > + if (ev != QEMU_PLUGIN_EV_ATEXIT) { > + struct qemu_plugin_ctx *ctx; > + QTAILQ_FOREACH(ctx, &plugin.ctxs, entry) { > + plugin_unregister_cb__locked(ctx, ev); > + } > + } > + } > + > + tb_flush(current_cpu); > + end_exclusive(); > + > + /* now it's safe to handle the exit case */ > + qemu_plugin_atexit_cb(); > +} > + > /* > * Call this function after longjmp'ing to the main loop. It's possible > that the > * last instruction of a TB might have used helpers, and therefore the > -- > 2.32.0.264.g75ae10bc75 > > <div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr">On Mon, Jul 19, 2021, 14:37 Alex Bennée <<a href="mailto:alex.bennee@linaro.org">alex.bennee@linaro.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">In user-mode emulation there is a small race between preexit_cleanup<br> and exit_group() which means we may end up calling instrumented<br> instructions before the kernel reaps child threads. To solve this we<br> implement a new helper which ensures the callbacks are flushed along<br> with any translations before we let the host do it's a thing.<br> <br> While we are at it make the documentation of<br> qemu_plugin_register_atexit_cb clearer as to what the user can expect.<br> <br> Signed-off-by: Alex Bennée <<a href="mailto:alex.bennee@linaro.org" target="_blank" rel="noreferrer">alex.bennee@linaro.org</a>><br></blockquote></div></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">Looks great to me and I cannot reproduce the race with it installed in.</div><div dir="auto"><br></div><div dir="auto">With calling `qemu_plugin_disable_mem_helpers`:</div><div dir="auto"><br></div><div dir="auto">Reviewed-by: Mahmoud Mandour <<a href="mailto:ma.mandourr@gmail.com">ma.mandourr@gmail.com</a>></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> ---<br> include/qemu/plugin.h | 12 ++++++++++++<br> include/qemu/qemu-plugin.h | 13 +++++++++++++<br> bsd-user/syscall.c | 6 +++---<br> linux-user/exit.c | 2 +-<br> plugins/core.c | 33 +++++++++++++++++++++++++++++++++<br> 5 files changed, 62 insertions(+), 4 deletions(-)<br> <br> diff --git a/include/qemu/plugin.h b/include/qemu/plugin.h<br> index 0fefbc6084..9a8438f683 100644<br> --- a/include/qemu/plugin.h<br> +++ b/include/qemu/plugin.h<br> @@ -190,6 +190,16 @@ void qemu_plugin_add_dyn_cb_arr(GArray *arr);<br> <br> void qemu_plugin_disable_mem_helpers(CPUState *cpu);<br> <br> +/**<br> + * qemu_plugin_user_exit(): clean-up callbacks before calling exit callbacks<br> + *<br> + * This is a user-mode only helper that ensure we have fully cleared<br> + * callbacks from all threads before calling the exit callbacks. This<br> + * is so the plugins themselves don't have to jump through hoops to<br> + * guard against race conditions.<br> + */<br> +void qemu_plugin_user_exit(void);<br> +<br> #else /* !CONFIG_PLUGIN */<br> <br> static inline void qemu_plugin_add_opts(void)<br> @@ -250,6 +260,8 @@ void qemu_plugin_add_dyn_cb_arr(GArray *arr)<br> static inline void qemu_plugin_disable_mem_helpers(CPUState *cpu)<br> { }<br> <br> +static inline void qemu_plugin_user_exit(void)<br> +{ }<br> #endif /* !CONFIG_PLUGIN */<br> <br> #endif /* QEMU_PLUGIN_H */<br> diff --git a/include/qemu/qemu-plugin.h b/include/qemu/qemu-plugin.h<br> index dc3496f36c..e6e815abc5 100644<br> --- a/include/qemu/qemu-plugin.h<br> +++ b/include/qemu/qemu-plugin.h<br> @@ -549,6 +549,19 @@ void qemu_plugin_vcpu_for_each(qemu_plugin_id_t id,<br> void qemu_plugin_register_flush_cb(qemu_plugin_id_t id,<br> qemu_plugin_simple_cb_t cb);<br> <br> +/**<br> + * qemu_plugin_register_atexit_cb() - register exit callback<br> + * @id: plugin ID<br> + * @cb: callback<br> + * @userdata: user data for callback<br> + *<br> + * The @cb function is called once execution has finished. Plugins<br> + * should be able to free all their resources at this point much like<br> + * after a reset/uninstall callback is called.<br> + *<br> + * In user-mode it is possible a few un-instrumented instructions from<br> + * child threads may run before the host kernel reaps the threads.<br> + */<br> void qemu_plugin_register_atexit_cb(qemu_plugin_id_t id,<br> qemu_plugin_udata_cb_t cb, void *userdata);<br> <br> diff --git a/bsd-user/syscall.c b/bsd-user/syscall.c<br> index 7d986e9700..3f44311396 100644<br> --- a/bsd-user/syscall.c<br> +++ b/bsd-user/syscall.c<br> @@ -335,7 +335,7 @@ abi_long do_freebsd_syscall(void *cpu_env, int num, abi_long arg1,<br> _mcleanup();<br> #endif<br> gdb_exit(arg1);<br> - qemu_plugin_atexit_cb();<br> + qemu_plugin_user_exit();<br> /* XXX: should free thread stack and CPU env */<br> _exit(arg1);<br> ret = 0; /* avoid warning */<br> @@ -437,7 +437,7 @@ abi_long do_netbsd_syscall(void *cpu_env, int num, abi_long arg1,<br> _mcleanup();<br> #endif<br> gdb_exit(arg1);<br> - qemu_plugin_atexit_cb();<br> + qemu_plugin_user_exit();<br> /* XXX: should free thread stack and CPU env */<br> _exit(arg1);<br> ret = 0; /* avoid warning */<br> @@ -516,7 +516,7 @@ abi_long do_openbsd_syscall(void *cpu_env, int num, abi_long arg1,<br> _mcleanup();<br> #endif<br> gdb_exit(arg1);<br> - qemu_plugin_atexit_cb();<br> + qemu_plugin_user_exit();<br> /* XXX: should free thread stack and CPU env */<br> _exit(arg1);<br> ret = 0; /* avoid warning */<br> diff --git a/linux-user/exit.c b/linux-user/exit.c<br> index 70b344048c..527e29cbc1 100644<br> --- a/linux-user/exit.c<br> +++ b/linux-user/exit.c<br> @@ -35,5 +35,5 @@ void preexit_cleanup(CPUArchState *env, int code)<br> __gcov_dump();<br> #endif<br> gdb_exit(code);<br> - qemu_plugin_atexit_cb();<br> + qemu_plugin_user_exit();<br> }<br> diff --git a/plugins/core.c b/plugins/core.c<br> index e1bcdb570d..c573b81a96 100644<br> --- a/plugins/core.c<br> +++ b/plugins/core.c<br> @@ -487,6 +487,39 @@ void qemu_plugin_register_atexit_cb(qemu_plugin_id_t id,<br> plugin_register_cb_udata(id, QEMU_PLUGIN_EV_ATEXIT, cb, udata);<br> }<br> <br> +/*<br> + * Handle exit from linux-user. Unlike the normal atexit() mechanism<br> + * we need to handle the clean-up manually as it's possible threads<br> + * are still running. We need to remove all callbacks from code<br> + * generation, flush the current translations and then we can safely<br> + * trigger the exit callbacks.<br> + */<br> +<br> +void qemu_plugin_user_exit(void)<br> +{<br> + enum qemu_plugin_event ev;<br> +<br> + QEMU_LOCK_GUARD(&plugin.lock);<br> +<br> + start_exclusive();<br> +<br> + /* un-register all callbacks except the final AT_EXIT one */<br> + for (ev = 0; ev < QEMU_PLUGIN_EV_MAX; ev++) {<br> + if (ev != QEMU_PLUGIN_EV_ATEXIT) {<br> + struct qemu_plugin_ctx *ctx;<br> + QTAILQ_FOREACH(ctx, &plugin.ctxs, entry) {<br> + plugin_unregister_cb__locked(ctx, ev);<br> + }<br> + }<br> + }<br> +<br> + tb_flush(current_cpu);<br> + end_exclusive();<br> +<br> + /* now it's safe to handle the exit case */<br> + qemu_plugin_atexit_cb();<br> +}<br> +<br> /*<br> * Call this function after longjmp'ing to the main loop. It's possible that the<br> * last instruction of a TB might have used helpers, and therefore the<br> -- <br> 2.32.0.264.g75ae10bc75<br> <br> </blockquote></div></div></div>
diff --git a/include/qemu/plugin.h b/include/qemu/plugin.h index 0fefbc6084..9a8438f683 100644 --- a/include/qemu/plugin.h +++ b/include/qemu/plugin.h @@ -190,6 +190,16 @@ void qemu_plugin_add_dyn_cb_arr(GArray *arr); void qemu_plugin_disable_mem_helpers(CPUState *cpu); +/** + * qemu_plugin_user_exit(): clean-up callbacks before calling exit callbacks + * + * This is a user-mode only helper that ensure we have fully cleared + * callbacks from all threads before calling the exit callbacks. This + * is so the plugins themselves don't have to jump through hoops to + * guard against race conditions. + */ +void qemu_plugin_user_exit(void); + #else /* !CONFIG_PLUGIN */ static inline void qemu_plugin_add_opts(void) @@ -250,6 +260,8 @@ void qemu_plugin_add_dyn_cb_arr(GArray *arr) static inline void qemu_plugin_disable_mem_helpers(CPUState *cpu) { } +static inline void qemu_plugin_user_exit(void) +{ } #endif /* !CONFIG_PLUGIN */ #endif /* QEMU_PLUGIN_H */ diff --git a/include/qemu/qemu-plugin.h b/include/qemu/qemu-plugin.h index dc3496f36c..e6e815abc5 100644 --- a/include/qemu/qemu-plugin.h +++ b/include/qemu/qemu-plugin.h @@ -549,6 +549,19 @@ void qemu_plugin_vcpu_for_each(qemu_plugin_id_t id, void qemu_plugin_register_flush_cb(qemu_plugin_id_t id, qemu_plugin_simple_cb_t cb); +/** + * qemu_plugin_register_atexit_cb() - register exit callback + * @id: plugin ID + * @cb: callback + * @userdata: user data for callback + * + * The @cb function is called once execution has finished. Plugins + * should be able to free all their resources at this point much like + * after a reset/uninstall callback is called. + * + * In user-mode it is possible a few un-instrumented instructions from + * child threads may run before the host kernel reaps the threads. + */ void qemu_plugin_register_atexit_cb(qemu_plugin_id_t id, qemu_plugin_udata_cb_t cb, void *userdata); diff --git a/bsd-user/syscall.c b/bsd-user/syscall.c index 7d986e9700..3f44311396 100644 --- a/bsd-user/syscall.c +++ b/bsd-user/syscall.c @@ -335,7 +335,7 @@ abi_long do_freebsd_syscall(void *cpu_env, int num, abi_long arg1, _mcleanup(); #endif gdb_exit(arg1); - qemu_plugin_atexit_cb(); + qemu_plugin_user_exit(); /* XXX: should free thread stack and CPU env */ _exit(arg1); ret = 0; /* avoid warning */ @@ -437,7 +437,7 @@ abi_long do_netbsd_syscall(void *cpu_env, int num, abi_long arg1, _mcleanup(); #endif gdb_exit(arg1); - qemu_plugin_atexit_cb(); + qemu_plugin_user_exit(); /* XXX: should free thread stack and CPU env */ _exit(arg1); ret = 0; /* avoid warning */ @@ -516,7 +516,7 @@ abi_long do_openbsd_syscall(void *cpu_env, int num, abi_long arg1, _mcleanup(); #endif gdb_exit(arg1); - qemu_plugin_atexit_cb(); + qemu_plugin_user_exit(); /* XXX: should free thread stack and CPU env */ _exit(arg1); ret = 0; /* avoid warning */ diff --git a/linux-user/exit.c b/linux-user/exit.c index 70b344048c..527e29cbc1 100644 --- a/linux-user/exit.c +++ b/linux-user/exit.c @@ -35,5 +35,5 @@ void preexit_cleanup(CPUArchState *env, int code) __gcov_dump(); #endif gdb_exit(code); - qemu_plugin_atexit_cb(); + qemu_plugin_user_exit(); } diff --git a/plugins/core.c b/plugins/core.c index e1bcdb570d..c573b81a96 100644 --- a/plugins/core.c +++ b/plugins/core.c @@ -487,6 +487,39 @@ void qemu_plugin_register_atexit_cb(qemu_plugin_id_t id, plugin_register_cb_udata(id, QEMU_PLUGIN_EV_ATEXIT, cb, udata); } +/* + * Handle exit from linux-user. Unlike the normal atexit() mechanism + * we need to handle the clean-up manually as it's possible threads + * are still running. We need to remove all callbacks from code + * generation, flush the current translations and then we can safely + * trigger the exit callbacks. + */ + +void qemu_plugin_user_exit(void) +{ + enum qemu_plugin_event ev; + + QEMU_LOCK_GUARD(&plugin.lock); + + start_exclusive(); + + /* un-register all callbacks except the final AT_EXIT one */ + for (ev = 0; ev < QEMU_PLUGIN_EV_MAX; ev++) { + if (ev != QEMU_PLUGIN_EV_ATEXIT) { + struct qemu_plugin_ctx *ctx; + QTAILQ_FOREACH(ctx, &plugin.ctxs, entry) { + plugin_unregister_cb__locked(ctx, ev); + } + } + } + + tb_flush(current_cpu); + end_exclusive(); + + /* now it's safe to handle the exit case */ + qemu_plugin_atexit_cb(); +} + /* * Call this function after longjmp'ing to the main loop. It's possible that the * last instruction of a TB might have used helpers, and therefore the
In user-mode emulation there is a small race between preexit_cleanup and exit_group() which means we may end up calling instrumented instructions before the kernel reaps child threads. To solve this we implement a new helper which ensures the callbacks are flushed along with any translations before we let the host do it's a thing. While we are at it make the documentation of qemu_plugin_register_atexit_cb clearer as to what the user can expect. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> --- include/qemu/plugin.h | 12 ++++++++++++ include/qemu/qemu-plugin.h | 13 +++++++++++++ bsd-user/syscall.c | 6 +++--- linux-user/exit.c | 2 +- plugins/core.c | 33 +++++++++++++++++++++++++++++++++ 5 files changed, 62 insertions(+), 4 deletions(-) -- 2.32.0.264.g75ae10bc75