Lab 06: Capstone
In this lab you bring your tests from Labs 04 and 05 together into one suite and use it to find the three tier-5 bugs, which are the hardest in the catalogue to think of. You finish with a suite that catches all thirteen bugs and a written bug report for one of the three. None of the three is subtle in the RTL. B10 and B12 are hard to catch because each one needs a test input that ordinary tests don’t happen to produce, and B11 is easy to catch on this design, for a reason Part 3 explains.
This lab takes about 3 hours 30 minutes.
Package: labs/lab-06-capstone/
Before you start
- Finish Lab 05: you have protocol tests as well as architectural ones.
- Read Writing the bug report, and reread Directed and random tests.
- Have your Lab 04 and Lab 05 test files at hand to bring forward.
The three bugs in this lab’s pool, and why each survives:
| Survives because | |
|---|---|
B10_ori_adds_instead_of_ors |
0 \| imm equals 0 + imm, so the first ORI after reset is always right, and OR and ADD agree exactly when the operands share no set bits, which is every constant anyone builds on purpose |
B11_ori_writes_r1_reads_r0 |
The value it produces is correct and only its destination is wrong, and right after a reset the two registers it could have written hold the same thing |
B12_uart_rx_bit_period |
It has no effect at all at the default parameter value, and the link test that everybody writes first passes even over the broken link |
Read the coverage list below as a set of questions to ask about your own suite. Each one has an answer you can measure:
| Ask | |
|---|---|
| Instructions | Which of the sixteen opcodes has my suite never executed, including the three unimplemented ones? |
| Operands | For each instruction, have I run it with rA == rB and with rA != rB? |
| Fields | Have I used every shift amount (4), every branch displacement (16), and every ORI immediate that overlaps bits already set? |
| Registers | Has each of the four been a source, a destination, a pointer, and a bystander that must not move? |
| Flags | Do I assert the flags after every flag-setting instruction, and assert that they don’t move after the ones that leave them alone? |
| Memory | Have I used address 0, address 63, and an address above 63? |
| Protocol | Do I count done, count halt, and assert cycle counts? |
| Parameters | Have I swept clk_per_bit, rather than only sampling it at its default? |
| State | Does anything in my suite run long enough to leave the window where reset has made everything equal? |
The last two rows are where B12 and B11 hide, and the Fields row is where B10 hides.
mod dd104 # in a new terminal, load .designinit first (Lab 01, Part 1)
cd labs/lab-06-capstonePart 1: Get a baseline
How many of the thirteen does the suite you already have catch?
Copy your Lab 04 and Lab 05 tests into
test_capstone.py.make check-allscores whatever that module contains, so copy them as they are; there is no credit for retyping work.Score it against all thirteen:
make check-allIt takes about four minutes, because it runs fourteen full builds and simulations. You should see a scoreboard and, for each bug you missed, a line saying what it hides behind:
caught tier 1 B01_sub_operands_swapped t01_arithmetic 20.9s ... MISSED tier 5 B12_uart_rx_bit_period ------------------------------------------------------------------------ 10 of 13 caught. Still hiding from this testbench: B12_uart_rx_bit_period hides behind: the default baud rate, where it is invisibleYour count and list will depend on what you brought in.
This is the only place in the course where a failure comes with a hint, so use the list of missed bugs as your work queue for the rest of the lab.
Check: you have a baseline count.
Part 2: B10, the wrong operator
Why does a good testbench miss an ORI that adds instead of ORs?
Before you run, predict whether the six-test example testbench catches
B10. Then, inlabs/common:cd ../common make check TB=test_example BUG=B10 cd ../lab-06-capstoneYou should see
MISSED B10_ori_adds_instead_of_orsand the hintORing in bits that were not already set — ADD agrees. The example testbench has six tests and a 29-instruction comparison against the model, and none of those instructions ORs in a bit that was already set.The hint says when
ADDagrees withOR. Before writing a test, work out why that makes this bug hard to catch: try5 | 3against5 + 3, then think about how a program on this machine builds a constant, and how often it ORs a bit into a register that already has it.TipAnswer5 | 3 = 7but5 + 3 = 8; then7 | 24 = 31but8 + 24 = 32. OR and ADD give the same answer on every pair of operands that share no set bits. Building a constant on this machine only ever ORs in bits that aren’t set yet, which is exactly the case where the two agree, and that is whatset_regdoes:SUB r0, r0 # clear ORI 0x16 # the top five bits SHL r0, 3 # shove them up ORI 0x07 # the bottom three -- disjoint from what is already thereEvery one of those
ORIs gives the right answer on a core whoseORIadds. The firstORIafter a reset is safe too, because it computes0 | imm, which equals0 + imm, so even an overlapping immediate gives the right answer the first time.Write a test that ORs a bit into a register that already has it:
await sp.run([ORI(5), ORI(3), ORI(24)]) assert sp.reg(0) == 31Run
make check BUG=B10. You should seecaught, because the register holds31on a correct core and32onB10.
In the eleven-test reference suite, 2 of the 11 tests catch B10.
Check: make check BUG=B10 says caught.
Part 3: B11, the wrong destination register
Why can a value be right and still be wrong?
See how the reference suite does against it:
make check TB=test_reference BUG=B11You should see:
caught B11_ori_writes_r1_reads_r0 11 tests, 7 FAILED 14.2s R01_alu_asymmetric_operands R02_ori_overlapping_immediates R03_shift_amount_sweep R05_flags_survive_a_branchSeven failing tests out of eleven is an exact count, and it’s the highest of the three bugs in this pool by some distance.
Compare what a five-instruction constant builder leaves in the registers on the two cores (both rows are exact):
r0r1Golden 20 0 B110 4 On
B11,ORIreadsr0but writesr1, so the shift in the middle of the builder shiftsr0, a register that never received anything.Make sure your suite catches it with
make check BUG=B11.Why is
B11easy to catch in this design, and why would a short test that only checks the value miss it?TipAnswerIt is easy to catch here because
ORIis the only way to get a constant into the machine, so everyset_regin every test goes through it (not every design is that obliging). A short test that only checks the value misses it because the value is right and only the destination is wrong. Right after a resetr0andr1hold the same thing, so a machine that reads one and writes the other produces a number that is correct in every respect except where it ends up. Reset is a state in which several things are temporarily indistinguishable, and a test that stays inside that window can’t tell them apart. Your programs are short because short programs are readable, and that is what keeps them inside the window.
B11 is also upstream’s own to-do, applied to only one of the two places it touches. The SimProc source shipped with a comment suggesting exactly this change, and we removed that comment from DD104’s fork on purpose, because in a course whose premise is “there are hidden bugs in this file” it reads as a planted clue pointing at code that is in fact correct. ORI-DECISION.md has the full reasoning, including why the shipped behaviour is the specified one. Neither a source comment nor a to-do is a specification, and B11 shows what happens when a to-do is applied halfway.
Check: your suite catches B11.
Part 4: B12, the baud rate
Does the UART work at every speed, or only at the default?
The UART’s bit period, clk_per_bit, is uio_in × 4. Every lab so far has run at the default uio_in = 8, which gives clk_per_bit = 32.
Write a test that sweeps the bit period and, at each speed, sends a
PINGand also writes and reads back a byte that is not a fixed pattern:for uio in (4, 8, 16, 32, 64): await sp.set_baud(uio) reply = await sp.uart_ping() await sp.uart_write(0x3F, 0x5A) got = await sp.uart_read(0x3F)B12adds one clock to every bit the UART receives. Before you run, predict whether it starts failing at short bit periods or at long ones. Then run the test withmake sim, and after that withmake check BUG=B12. On the golden core every speed works, and onB12you should see this pattern (every row is exact):clk_per_bitPINGA data byte ( 0x5A, written then read back)8 TimeoutError, no reply at all16 0xAAValueError, because the reply is X bits24 0xAA0x5A32 (the labs’ default) 0xAA0x5A64, 128, 256 0xAA0x5AAt
clk_per_bit = 16,PINGgets a perfect0xAAreply over a link that cannot carry a data byte. Why?TipAnswerIt’s because of the particular bytes involved. The
PINGcommand,0xF0, already has its top bit set, and the reply0xAAis a fixed pattern, so a receiver that mis-samples bit 7 as the stop bit still gets both of them right. A link test that passes over a broken link is worse than having no link test, because you believe it.Why does
B12fail at short bit periods first, and look correct at 24 and above?TipAnswerThe extra clock per received bit makes the sample point drift further off centre with each bit of a frame, and a long bit period has room to absorb that drift. At the default speed this core behaves like a correct one, and the rest of your suite scores the same on it as on the golden core, so the pattern of which speeds work is your diagnosis. Only 1 of the 11 tests in the reference suite catches
B12, and it’s the only one that sweeps the bit period. When a design has a parameter, test it at more than one value.
Check: make check BUG=B12 says caught.
Part 5: Hunt one properly
Can you find which of the three your dealt core has?
Deal yourself a core with one of the three bugs and hunt for it:
make new make hunt make verdictThe grade is the same three rows as in Lab 04, and row 2 is still the deliverable. The Lab 04 tools are here too:
make probe, which shows everyprint()anddut._logline as it happens, andmake hunt VERBOSE=1andmake verdict VERBOSE=1.
Knowing that three bugs exist doesn’t tell you which one you have. The pool description in test_capstone.py describes all three without naming the line any of them touches, which is about as much as a bug report tells you in real life.
Check: make verdict says VERDICT: PASS.
Part 6: The suite
Does your suite catch all thirteen, and which tests do the catching?
Run
make check-alluntil it says:------------------------------------------------------------------------ 13 of 13 caught. Every bug in the catalogue, every one of them caught by a check that fired. Nothing in this course hides from this testbench any more. ========================================================================Look at which test catches each bug. In the reference capstone suite, the catches are:
caught tier 1 B01_sub_operands_swapped t01_arithmetic caught tier 1 B02_nand_is_nor t01_arithmetic caught tier 1 B03_zero_flag_low_nibble t08_against_the_golden caught tier 2 B04_alu_writeback_reg_field t01_arithmetic caught tier 2 B06_shift_amount_one_bit t01_arithmetic caught tier 2 B07_load_address_from_ra t07_load_through_a_sep caught tier 3 B05_branch_offset_zero_extended t04_a_loop caught tier 3 B09_branch_clobbers_flags t10_a_compare_survives caught tier 4 B08_store_drops_done p01_done_pulses_once_p caught tier 4 B13_store_falls_to_idle t04_a_loop caught tier 5 B10_ori_adds_instead_of_ors t08_against_the_golden caught tier 5 B11_ori_writes_r1_reads_r0 t01_arithmetic caught tier 5 B12_uart_rx_bit_period c03_baud_sweepThe reference suite has sixteen tests: ten from Lab 04, three from Lab 05 and three new ones. Most of the work was already done before this lab began, but the three new tests took the longest.
Two general tests,
t01_arithmeticandt08_against_the_golden_model, catch seven of the thirteen between them, and five bugs are caught by exactly one test each. Is that a well-balanced suite, or one with a long tail of special cases bolted on? Write down your answer and what you would write next. Either reading can be defended.
Check: 13 of 13 caught.
Part 7: Practise the report on B01
Can you take a bug you already know through the whole report loop?
Run the worked reproducer in
labs/common:cd ../common make check TB=test_b01_repro BUG=B01You should see
okon the golden core andcaught B01_sub_operands_swapped 3 tests, 1 FAILED, with the messageSUB r1, r2 with r1 = 200 and r2 = 100 left r1 = 156.Deal
B01in the Lab 04 package and grade the reproducer against it.make newdeletes thecore/already inlabs/lab-04-your-first-bug-hunt, so finish anything you still want from your Lab 04 core first.cd ../lab-04-your-first-bug-hunt make new BUGPICK=B01 make verdict TB=test_b01_reproThe
make newbanner still lists Lab 04’s five-bug pool, but the core it wrote hasB01. You should see rows 1 and 2 green and row 3FAIL, which means the test exists and the design is still broken.ok 1. the golden core must PASS 3 tests, all passed 7.8s caught 2. the core you were dealt must FAIL 3 tests, 1 FAILED 4.2s t01_sub_with_two_distinct_operands AssertionError: SUB r1, r2 with r1 = 200 and r2 = 100 left r1 = 156. FAIL 3. core/ as it stands now must PASS 3 tests, 1 FAILED 4.1sFix line 811 of
core/simproc_system.sv:SUB: ALUout = B - A; -> SUB: ALUout = A - B;Grade it again, with the full failure message shown:
make verdict TB=test_b01_repro VERBOSE=1You should see
VERDICT: PASSwith row 2 unchanged, and the whole assertion message from Writing the bug report, includingnote (100 - 200) mod 256 = 156.Go back to the capstone package:
cd ../lab-06-capstone
Check: row 2 stayed caught after the fix.
Part 8: Write the report
Can someone act on your bug without redoing your work?
Copy the template:
cp BUG-REPORT-template.md BUG-REPORT.mdFill it in for one of
B10,B11andB12, usingBUG-REPORT-EXAMPLE.mdas a model. Five parts:- The smallest program that reproduces it. Cut instructions until it stops failing, then put the last one back.
- Expected against actual, as numbers, with the arithmetic shown.
- The shape: how many vectors or cycles are wrong, out of how many, and what the count implies before anyone opens the RTL.
- Where it isn’t: what still works.
- The habit the suite gained, in one line, and the test that enforces it, by name.
A red test is only where the report starts; the report itself is what lets someone act on the bug without redoing your week. Part 4 of the report is the one people skip, and it’s the one designers read first, because a sentence like “every ORI whose immediate is disjoint from r0 is correct, all 393,216 ALU vectors are correct, and both branches and all memory accesses are unaffected” tells them where not to look, and that turns a report into a map. The report is finished when a test in the suite fails before the fix and passes after it, which is what make verdict checks.
Check: BUG-REPORT.md exists and has all five parts.
Part 9: Close the course
What claim does your suite now support?
Write down, in one sentence, the claim your suite now supports. Start it with “This processor…” and keep going until the sentence is true.
Put it next to the claim the chip shipped with:
assert dut.uo_out.value == 3which supports only “this processor is not running”. How long is your sentence, and why?
TipAnswerIt should be uncomfortably long. It will have clauses about operands that are not equal to each other, about every value of a two-bit field and both halves of a four-bit signed one, about flags that move and flags that must not, about a pin that pulses once per instruction and a pin that never rises, about an exact number of clock cycles, and about a serial link at five different speeds. Each clause is there because a bug got past you first. A test suite is a way to make a long, specific, checkable claim, and you write one because nobody, including you next year, can hold a claim that long in their head.
The part of this course you will use everywhere is the question to ask of any green test, including your own: what would this have caught?
Check: you have your sentence written next to the shipped claim.
Expected results
| Check | Expected | |
|---|---|---|
make check-all |
13 of 13 caught |
exact |
make check-all wall clock |
about 4 min 35 s | ballpark |
| Reference suite on golden | 16 of 16, 2118170.02 ns |
exact |
Tests failing, reference 11-test suite: B10 / B11 / B12 |
2 / 7 / 1 | exact |
B10: ORI 5, ORI 3, ORI 24 |
golden 31, broken 32 |
exact |
B11: five-instruction constant builder |
golden r0 = 20; broken r0 = 0, r1 = 4 |
exact |
B12 at clk_per_bit = 16 |
PING returns 0xAA; a data byte does not survive |
exact |
B12 at clk_per_bit >= 24 |
indistinguishable from a correct core | exact |
BUG-REPORT.md |
exists, five parts |
Finishing
You’re done when make check-all on your test_capstone.py says 13 of 13 caught and your BUG-REPORT.md has all five parts.
Submit: test_capstone.py and BUG-REPORT.md.
Going further (optional)
Read labs/common/test_reference.py, which catches all thirteen in eleven tests, and find the three choices you disagree with.
Next on the Course path: this is the last lab. The digital track continues with DD201 (High-Level Synthesis) and DD202 (Computer Architecture).