Testing instructions / Reproducible builds


#1

I am unable to successfully test the repo locally. It would be useful to include instructions on how to test the repo including (toolchain, command, other dependencies).

Here are some questions:

  • are we supposed to use stable to test? (maybe add a rust-toolchain file)
  • how do we invoke all the tests (I am using cargo test --all). but this could be different if someone is adding benchmarking tests
  • are there additional deps needed to test the repo?

my toolchain

rustup show active-toolchain
beta-x86_64-apple-darwin

Here is the error I am getting on both master and next branches:

cmd: RUST_BACKTRACE=1 cargo test --all

...

Running target/debug/deps/rendering-85d74b747276a92b

running 17 tests
test shortcode::tests::lex_inline_ignored_shortcode ... ok
test shortcode::tests::can_parse_shortcode_arguments ... ok
test shortcode::tests::lex_ignored_shortcode_with_body ... ok
test shortcode::tests::lex_inline_shortcode ... ok
test shortcode::tests::lex_text ... ok
test shortcode::tests::lex_shortcode_with_body ... ok
test shortcode::tests::lex_page ... ok
test table_of_contents::tests::can_make_basic_toc ... ok
test table_of_contents::tests::can_make_messy_toc ... ok
test table_of_contents::tests::can_make_more_complex_toc ... ok
test shortcode::tests::can_unignore_inline_shortcode ... ok
test shortcode::tests::does_nothing_with_no_shortcodes ... ok
test shortcode::tests::unignore_shortcode_with_body_does_not_swallow_initial_whitespace ... ok
test shortcode::tests::can_unignore_shortcode_with_body ... ok
test shortcode::tests::can_render_shortcodes_with_body ... ok
test shortcode::tests::can_render_inline_shortcodes ... ok
test shortcode::tests::shortcodes_with_body_do_not_eat_newlines ... ok

test result: ok. 17 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/markdown-ade83266ef369348

running 41 tests
test can_add_id_to_headers_same_slug ... ok
test can_handle_summaries ... ok
test can_do_render_content_simple ... ok
test can_add_id_to_headers ... ok
test can_insert_anchor_left ... ok
test can_insert_anchor_with_link ... ok
test can_insert_anchor_right ... ok
test can_insert_anchor_with_exclamation_mark ... ok
test can_ignore_tags_in_toc ... ok
test can_make_permalinks_with_colocated_assets_for_image ... ok
test can_insert_anchor_with_other_special_chars ... ok
test can_make_permalinks_with_colocated_assets_for_link ... ok
test can_make_relative_links_with_anchors ... ok
test can_make_toc ... ok
test can_make_valid_relative_link ... ok
test can_make_valid_relative_link_in_header ... ok
test can_render_body_shortcode_and_paragraph_after ... ok
test can_render_shortcode ... ok
test can_render_shortcode_with_body ... ok
test can_render_body_shortcode_with_markdown_char_in_name ... ok
test can_render_several_shortcode_in_row ... ok
test can_render_shortcode_with_markdown_char_in_args_name ... ok
test can_understand_backtick_in_paragraphs ... ok
test can_render_two_body_shortcode_and_paragraph_after_with_line_break_between ... ok
test can_understand_backtick_in_titles ... ok
test can_render_shortcode_with_markdown_char_in_args_value ... ok
test can_understand_link_with_title_in_header ... ok
test can_understand_links_in_header ... ok
test doesnt_highlight_code_block_with_highlighting_off ... ok
test doesnt_render_ignored_shortcodes ... ok
test doesnt_try_to_validate_email_links_angled_brackets ... ok
test doesnt_try_to_validate_email_links_mailto ... ok
test errors_relative_link_inexistant ... ok
test markdown_doesnt_wrap_html_in_paragraph ... ok
test errors_rendering_unknown_shortcode ... ok
test doesnt_try_to_highlight_content_from_shortcode ... ok
test can_validate_valid_external_links ... ok
test can_highlight_code_block_no_lang ... ok
test can_higlight_code_block_with_unknown_lang ... ok
test can_highlight_code_block_with_lang ... ok
test can_show_error_message_for_invalid_external_links ... FAILED

failures:

---- can_show_error_message_for_invalid_external_links stdout ----
thread 'can_show_error_message_for_invalid_external_links' panicked at 'assertion failed: res.is_err()', components/rendering/tests/markdown.rs:634:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:211
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:221
   4: <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get
             at src/libstd/panicking.rs:491
   5: core::ptr::real_drop_in_place
             at /rustc/eb53c1039300a8bc8f8e4b29659ddc80b1e81b68/src/libstd/panicking.rs:425
   6: markdown::can_show_error_message_for_invalid_external_links
             at components/rendering/tests/markdown.rs:634
   7: markdown::can_show_error_message_for_invalid_external_links::{{closure}}
             at components/rendering/tests/markdown.rs:622
   8: core::ops::function::FnOnce::call_once
             at /rustc/eb53c1039300a8bc8f8e4b29659ddc80b1e81b68/src/libcore/ops/function.rs:238
   9: <F as alloc::boxed::FnBox<A>>::call_box
             at src/libtest/lib.rs:1471
             at /rustc/eb53c1039300a8bc8f8e4b29659ddc80b1e81b68/src/libcore/ops/function.rs:238
             at /rustc/eb53c1039300a8bc8f8e4b29659ddc80b1e81b68/src/liballoc/boxed.rs:673
  10: panic_unwind::dwarf::eh::read_encoded_pointer
             at src/libpanic_unwind/lib.rs:102


failures:
    can_show_error_message_for_invalid_external_links

test result: FAILED. 40 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

#2

That is one of the two tests that actually does some HTTP requests, for the URL validation. However, this one souldn’t fail since the URL tested http://google.comy doesn’t exist and it doesn’t fail on travis OSX.

are we supposed to use stable to test? (maybe add a rust-toolchain file)

It works on stable yes, although next branch now requires 1.31

how do we invoke all the tests (I am using cargo test --all). but this could be different if someone is adding benchmarking tests

cargo test --all is the way to do it

are there additional deps needed to test the repo?

Nothing else


#3

@toidiu is the error happening 100% of the time?

This test is using reqwest but it has no requirements on OSX so not sure what’s going on.


#4

@keats sry I havnt had time to come back to this with New Years and then starting work again. I am hoping to get to this either this weekend or next week.


#5

The error is reproducible. Here is a more detailed output after commenting out is_err()

cmd: RUST_BACKTRACE=full cargo test --all can_show_error_message_for_invalid_external_links

---- can_show_error_message_for_invalid_external_links stdout ----
thread 'can_show_error_message_for_invalid_external_links' panicked at 'called `Result::unwrap_err()` on an `Ok` value: Rendered { body: "<p><a href=\"http://google.comy\">a link</a></p>\n", summary_len: None, toc: [] }', src/libcore/result.rs:999:5
stack backtrace:
   0:        0x10ab56973 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::hf75b4aa1b74bced2
                               at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1:        0x10ab5233c - std::sys_common::backtrace::_print::hb20aa9238779b092
                               at src/libstd/sys_common/backtrace.rs:70
   2:        0x10ab54fd1 - std::panicking::default_hook::{{closure}}::h7b94794f7ab864a0
                               at src/libstd/sys_common/backtrace.rs:58
                               at src/libstd/panicking.rs:200
   3:        0x10ab54b38 - std::panicking::default_hook::h775e7d1ba6fc7a3a
                               at src/libstd/panicking.rs:209
   4:        0x10ab5575e - <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get::hff3970dd641046fe
                               at src/libstd/panicking.rs:478
   5:        0x10ab5527c - std::panicking::continue_panic_fmt::h024f4a08366830b2
                               at src/libstd/panicking.rs:385
   6:        0x10ab55168 - std::panicking::try::do_call::h2c77af4886567476
                               at src/libstd/panicking.rs:312
   7:        0x10ab6cae1 - core::char::methods::<impl char>::escape_debug::hfadc34325eb6a812
                               at src/libcore/panicking.rs:85
   8:        0x109fea00f - core::result::unwrap_failed::ha16f4cc0e100402c
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libcore/macros.rs:16
   9:        0x109fe9c26 - <core::result::Result<T, E>>::unwrap_err::ha833d766b2e546c0
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libcore/result.rs:857
  10:        0x109fe5b39 - markdown::can_show_error_message_for_invalid_external_links::h26afd51844a2633f
                               at components/rendering/tests/markdown.rs:635
  11:        0x109fe9820 - markdown::can_show_error_message_for_invalid_external_links::{{closure}}::h657b58bc450c0a42
                               at components/rendering/tests/markdown.rs:622
  12:        0x109fe6e90 - core::ops::function::FnOnce::call_once::h390d4183abd765ba
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libcore/ops/function.rs:231
  13:        0x109ff22f1 - <F as alloc::boxed::FnBox<A>>::call_box::h3685e4730740a20c
                               at src/libtest/lib.rs:1475
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libcore/ops/function.rs:231
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/liballoc/boxed.rs:724
  14:        0x10ab58a7e - panic_unwind::dwarf::eh::read_encoded_pointer::h004141a04c2fc9a0
                               at src/libpanic_unwind/lib.rs:92
  15:        0x10a00e736 - test::run_test::run_test_inner::{{closure}}::hb092001af32bc651
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libstd/panicking.rs:276
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libstd/panic.rs:388
                               at src/libtest/lib.rs:1430
  16:        0x109fead74 - std::sys_common::backtrace::__rust_begin_short_backtrace::h2a69cef470afb5ec
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libstd/sys_common/backtrace.rs:135
  17:        0x109feb374 - std::panicking::try::do_call::h451b0b57824fb597
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libstd/thread/mod.rs:469
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libstd/panic.rs:309
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libstd/panicking.rs:297
  18:        0x10ab58a7e - panic_unwind::dwarf::eh::read_encoded_pointer::h004141a04c2fc9a0
                               at src/libpanic_unwind/lib.rs:92
  19:        0x109ff23c4 - <F as alloc::boxed::FnBox<A>>::call_box::h76fa35627c635563
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libstd/panicking.rs:276
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libstd/panic.rs:388
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/libstd/thread/mod.rs:468
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/liballoc/boxed.rs:724
  20:        0x10ab5816b - std::sys::unix::thread::Thread::new::thread_start::hc651d155b46de6bd
                               at /rustc/c0bbc3927e28c22edefe6a1353b5ecc95ea9a104/src/liballoc/boxed.rs:734
                               at src/libstd/sys_common/thread.rs:14
                               at src/libstd/sys/unix/thread.rs:81
  21:     0x7fffdfec193a - _pthread_body
  22:     0x7fffdfec1886 - _pthread_start


failures:
    can_show_error_message_for_invalid_external_links

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 40 filtered out

error: test failed, to rerun pass '--test markdown'


#6

Can you pretty print the value of res before the assert? I’m stumped x)


#7

So I tried it on another mac and can’t reproduce the error so I am going to mark this closed.

On a kind of related note I do an intermittent error with can_rebuild_after_editing_in_colocated_asset_folder_with_language. I suspect there might be a race condition with building the test site… I briefly tried to look into it.


#8

Yep, I think I fixed that one with the latest commit yesterday evening