Lab 02: The only test this chip has
You’ll run SimProc’s one shipped test against the good core and all thirteen broken ones and watch every one pass. Then you’ll drive the processor from Python, try the harness’s example testbench, compare Icarus with Verilator, and write a test that passes on the good core and fails on the one whose SUB has its operands swapped.
This lab takes about 1 hour 45 minutes.
Package: labs/lab-02-the-only-test-this-chip-has/. It includes ../common/lab.mk, which lives outside the package, so you need the whole repository checked out and not just this directory. Every lab from here on works the same way.
Before you start
- Finish Lab 01 (
make checksaysVERDICT: PASS) and cocotb on a counter. - Read Two simulators and SimProc in one page.
- Keep The SimProc instruction set and The SimProc harness open in other tabs.
In this lab, “the golden core” is the correct SimProc, and the thirteen broken ones are named B01 to B13. Work in a terminal on the noVNC desktop, where mod dd104 takes you to the course folder, as in Lab 01, Part 1:
mod dd104 # in a new terminal, load .designinit first (Lab 01, Part 1)
cd labs/lab-02-the-only-test-this-chip-hasPart 1: The shipped test
Which of fourteen processors does the shipped test catch, and what does it actually assert?
Run the shipped test against fourteen processors:
make shuttleThis takes about half a minute, because it builds and runs fourteen cores.
Before your first result you will see a wall of lines like this:
rtl/simproc_system.sv:552: sorry: constant selects in always_* processes are not fully supported (the process will be sensitive to all bits in 'instr_reg_out[7:0]').Icarus prints 28 of these every time it compiles the design, and it will keep doing so for the rest of the course. They’re notes rather than errors, and the exit code is 0; a real Icarus failure starts a line with
error:and stops. Output that is not an error explains why the simulation is still correct, and lists the other harmless messages like this one.You should see:
======================================================================== THE TEST THIS CHIP SHIPPED WITH ======================================================================== test/test.py, from the SimProc repository, untouched. It sets ui_in = 1, waits 100 clock cycles, and asserts uo_out == 3. Decoded: uart_tx = 1 (idle), halt = 1 (IDLE), done = 0. It asserts that the processor is NOT RUNNING. ------------------------------------------------------------------------ PASS the golden core 5.7s PASS B01_sub_operands_swapped 2.1s PASS B02_nand_is_nor 2.1s PASS B03_zero_flag_low_nibble 2.0s PASS B04_alu_writeback_reg_field 2.0s PASS B06_shift_amount_one_bit 2.0s PASS B07_load_address_from_ra 2.0s PASS B05_branch_offset_zero_extended 2.0s PASS B09_branch_clobbers_flags 2.0s PASS B08_store_drops_done 2.1s PASS B13_store_falls_to_idle 2.1s PASS B10_ori_adds_instead_of_ors 2.0s PASS B11_ori_writes_r1_reads_r0 2.0s PASS B12_uart_rx_bit_period 2.0s ------------------------------------------------------------------------ 13 of 13 deliberately broken processors pass the only test this chip has. ========================================================================13 of 13and the names are exact, but the seconds are wall-clock time and will vary. The cores are listed in tier order rather than by number. Each of the thirteen broken cores is a one-line edit tosimproc_system.sv.Open the test itself,
../common/test_shuttle.py. It’stest/test.pyfrom the SimProc repository, unchanged apart from a comment block that explains the pin decode.Write down, in one sentence, the strongest true claim it supports. (This is the first check-yourself question on What a passing test proves.)
TipAnswerAfter reset and 100 clock cycles with
ui_in[0]high, this design drivesuo_outto0x03. That’s a real claim, and even a useful one: it would catch a core that came up running by accident, or a UART that sent garbage at power-on. It’s just a very small claim, and nothing about the way the test is written tells you how small it is.Print the bug catalogue:
make bugsYou should see thirteen names in five tiers (a tier is how hard the bug is to think of), each with what it hides behind and what it teaches. It doesn’t tell you what any of them changes in the design.
Check: 13 of 13 on your screen, and a one-sentence claim written down.
Part 2: Make the chip do something
What does it take to make the processor run one instruction and check the result?
Run the lab’s testbench, which ships working:
make simYou should see:
** test_lab02.t01_the_test_this_chip_shipped_with PASS 1150.00 ** ** test_lab02.t02_what_that_number_meant PASS 1160.00 ** ** test_lab02.t03_one_instruction_that_actually_runs PASS 33300.00 ** ** TESTS=3 PASS=3 FAIL=0 SKIP=0 35610.00 **and, in the log,
uo_out = 3, and the core is IDLE, thenhalt=1 done=0 state=IDLE — this is a processor doing nothing, and t01 asserted exactly that, thenone ORI, 5 clock cycles, r0 = 5. The four simulated times and the5 clock cyclesshould match yours exactly.Read
t03intest_lab02.py. It takes only three lines of Python to make the processor run an instruction, and a fourth to check the result.sp.run()loads the program, adds a parking loop after it (aJMP -1, because the processor has no halt instruction), runs it, and returns once it reaches that loop:sp = SimProc(dut) await sp.start() # clock, reset, RAM blanked await sp.run([ORI(5)]) # load a program, run it to its parking loop assert sp.reg(0) == 5Compare
t01andt02. They assert the same thing about the same idle chip. Ist02a stronger test?TipAnswerNo. Both are exactly as weak as the shipped test, because both check the same idle chip. What
t02improves is the message it gives when it fails, which names the pin instead of handing you an integer to decode. Asserting on the decoded meaning rather than the bus value costs nothing, so do it.
The docstring at the top of test_lab02.py is a two-page summary of the harness API.
Check: TESTS=3 PASS=3 and 35610.00 ns.
Part 3: The example testbench
In how many different ways can a testbench catch one bug?
The harness directory, labs/common, ships six worked tests in test_example.py, and the intended way to start a testbench of your own is to copy that file.
Run the example against the golden core and
B01:cd ../common make checkmake checkusesBUG=B01unless you name another core. You should seeokon the golden core andcaughtonB01_sub_operands_swapped, with four tests failing in four different ways (long messages are cut to fit the table, and yours will be cut the same way):ok the golden core 6 tests, all passed 9.6s caught B01_sub_operands_swapped 6 tests, 4 FAILED 7.3s t03_arithmetic_on_different_values AssertionError: r1 is 199, expected 247. t04_flags_are_state AssertionError: after r1 = 255 the flags are N=0 Z=0, expected N=1 Z=0 t05_a_loop_and_its_cycle_count TimeoutError: the program did not reach its parking loop at address 16 within 20000 clock t06_against_the_golden_model AssertionError: instruction 19, at address 19 (0x66 = SUB r1, r2), left the machine in a ------------------------------------------------------------------------ VERDICT: PASSMatch each failure to drive, observe or decide, using the table in What a testbench is. Why does
t05time out instead of reporting a wrong value?TipAnswert05runs a loop controlled by a subtraction. OnB01the subtraction goes the wrong way, so the loop never ends and the program never reaches its parking loop. That’s why the symptom is a timeout rather than a mismatch, and it’s a useful bug report only because the test sets a deadline.Make your own copy and run it both ways:
cp test_example.py test_mine.py make sim TB=test_mine make check TB=test_mine BUG=B01TBtakes the name of a Python module rather than a file, so writeTB=test_mine, neverTB=test_mine.py. You should see six tests pass undermake sim, andmake checkshould give the samecaughtverdict as in step 1, this time under the nametest_mine.
Check: make check in labs/common ends on VERDICT: PASS.
Part 4: The golden model on its own
Can you predict a program’s exact cycle count from the instruction costs?
Still in
labs/common, run the golden model with no simulator. It runs the demo program built intogolden.py, a loop that sums 1 to 5 and stores the result at0x20:make modelYou should see:
16 instructions in memory, 28 executed, 109 clock cycles registers: r0=32 r1=15 r2=1 r3=32 mem[0x20] = 15 (all 28 instructions executed)Derive the 109 by hand, using the program listing that
make modelprints above those three lines and the cycle-count table in SimProc in one page: seven setup instructions, a three-instruction loop run five times, and six to finish.t05_a_loop_and_its_cycle_countintest_example.pyasserts the same number on the simulated hardware.TipAnswerSetup:
4+5+4+4+4+5+4 = 30. The loop body costs4+4+3 = 11and runs five times, for 55. The finish costs4+5+4+4+4+3 = 24. \(30 + 55 + 24 = 109\).
Check: your sum is 109.
Part 5: Two simulators
Do Icarus and Verilator agree about this design?
You stay in labs/common for this part.
Lint the design:
verilator --lint-only --timescale 1ns/1ps --top-module tb \ rtl/tt_um_ieeeuoftasic_simproc.sv rtl/simproc_system.sv rtl/tb.vYou should see twelve warnings in well under a second, including
rtl/simproc_system.sv:848:16: Bit extraction of array[63:0] requires 6 bit index, not 8 bits., ending with%Error: Exiting due to 12 warning(s). That last line is expected, because Verilator treats warnings as fatal, and here the warnings are what you ran it to find. What does line 848 tell you about addresses 64 to 255?TipAnswerThe RAM, module
dp_raminsimproc_system.sv, has 64 entries and the address bus is 8 bits, so addresses 64–255 are off the end of the array. Verilator found that just by reading the file, before simulating anything.Read past the end of the RAM on each simulator. Before you run, write down what you expect each one to return for address 100.
make clean && make sim TB=test_two_simulators SIM=icarus make clean && make sim TB=test_two_simulators SIM=verilatorYou should see
READ 100 (off the end) -> XXXXXXXXfrom Icarus andREAD 100 (off the end) -> 01011010from Verilator, withREAD 36giving01011010on both,TESTS=2 PASS=2and60520.00ns on both. Explain the Verilator value using100 & 63.TipAnswerVerilator is 2-state, so it truncates the address to the six bits the array has:
100 & 63 = 36, and you get the byte at address 36, which is the0x5Athe test wrote there before reading. Icarus is 4-state, so reading past the end givesx, and eightXcharacters come off the pin.Run the example testbench on both simulators and check that simulated time is identical, test by test:
make clean && make sim TB=test_example SIM=icarus >/dev/null 2>&1 && cp build/results.xml /tmp/ic.xml make clean && make sim TB=test_example SIM=verilator >/dev/null 2>&1 && cp build/results.xml /tmp/vl.xml diff <(grep -o 'sim_time_ns="[^"]*"' /tmp/ic.xml) \ <(grep -o 'sim_time_ns="[^"]*"' /tmp/vl.xml) && echo "identical -- every test, both simulators"You should see no diff output, then
identical -- every test, both simulators.Go back to the lab directory:
cd ../lab-02-the-only-test-this-chip-has
Check: the two simulators disagree about address 100 and agree on every simulated time.
Part 6: A waveform
What does one ORI look like inside the core?
Dump a waveform of the lab’s three tests:
make wavesIt takes about two seconds and writes
waves.vcd, which is 337,999 bytes. The size is exact, because the file is a dump of a deterministic simulation.Open it from your terminal on the noVNC desktop:
surfer waves.vcd # or: gtkwave waves.vcdAdd
clkanduo_out. Then opentb→tt_um_ieeeuoftasic_simproc→U1→CPU1and addcurr_state. The trace holds all three tests:t03runs from 2,310 ns to 35,610 ns, and itsORIcomes after the UART commands that start the core.You should see
curr_statewalk fromIDLEthroughCYCLE_1…CYCLE_5and back, withdone(bit 2 ofuo_out) pulsing as the instruction retires. It’s worth learning to move around the viewer now, while the trace is only 35,610 ns long, because later traces run to millions of nanoseconds.
Check: you can point at the five cycles of the ORI.
Part 7: Write a test that catches B01
Can you write a test that passes on the golden core and fails on the core whose SUB computes B - A?
Before running anything, decide: would
t03, which runsORI 5and checksr0 == 5, catch a processor whose subtractor has its operands backwards?Run the two-sided check:
make checkYou should see:
ok the golden core 3 tests, all passed 2.2s MISSED B01_sub_operands_swapped 3 tests, all passed 2.2s ------------------------------------------------------------------------ VERDICT: FAIL — your testbench passes B01_sub_operands_swapped too. A test that passes on a broken core is not testing that core. That bug hides behind: SUB rX, rX — the idiom for 'clear a register' Write a test whose stimulus does not do that. ========================================================================followed by
make: *** [../common/lab.mk:66: check] Error 1, which is make reporting that the verdict exited non-zero. Your testbench missesB01becauset03never subtracts.Read the hint.
SUB rX, rXis how you clear a register, so it appears in nearly every program on this machine. For how many of the 65,536 operand pairs doA - BandB - Agive the same byte?TipAnswerFor 512 of them. They agree for every pair with
A == B(256 pairs,SUB rX, rXamong them), and for every pair whose difference is 128, because 128 is its own negation in eight bits (256 more). That’s only 0.78 % of the input space, but the most natural subtraction test lands inside it.Do the TODO in
t03’s docstring: add a new test function that subtracts two registers holding different values and checks the result.set_reg(1, 200) + set_reg(2, 60) + [SUB(1, 2)]leavesr1at \(200 - 60 = 140\) on a correct core. Work out what a swapped subtractor gives before you run it.test_lab02.pyalready importsSUBandset_reg; a test has this shape:@cocotb.test() async def t04_subtract_two_different_numbers(dut): sp = SimProc(dut) await sp.start() await sp.run(...) # your program assert ... # with a message that names both numbersTipAnswerr1ends up at 116. A swapped subtractor computes \(60 - 200 = -140\), and \(-140\) in eight bits is \(116\).Assert the flags too:
sp.flags()returns(N, Z). PredictNandZafter200 - 60before you write the assertion.TipAnswerN = 1,Z = 0. \(140 = 0b10001100\), so bit 7 is set. The machine has no opinion about whether that byte means 140 or −116, becauseNis simply bit 7 of the result. See Flags as architectural state.Run
make checkagain. You should see:ok the golden core 4 tests, all passed 3.6s caught B01_sub_operands_swapped 4 tests, 1 FAILED 3.2s t04_subtract_two_different_numbers AssertionError: r1 is 116, expected 140 (200 - 60). ------------------------------------------------------------------------ VERDICT: PASSThe test name and the message will be whatever you wrote, but
140and116are exact. If you asserted the wrongN, the golden-core row readsFAILinstead and the verdict saysVERDICT: FAIL — your testbench does not pass the GOLDEN core.That’s the right order for things to go wrong in, because a testbench that fails on the golden core is telling you about a mistake in the testbench itself.
Check: make check says ok on the golden core and caught on B01, with your new test asserting both r1 and the flags.
Part 8: The scoreboard
How many of the thirteen does your testbench catch so far?
Run your testbench against all thirteen broken cores:
make check-allIt takes about 22 seconds and prints, for each core you missed, one line about what it hides behind. Keep the output, because it’s your scoreboard for the rest of the course: by Lab 06 the goal is
13 of 13 caught.On the shipped file the count is
1 of 13 caught, and the one it catches is:caught tier 5 B11_ori_writes_r1_reads_r0 t03_one_instruction_th 2.2sWith your Part 7 test added,
B01is caught too; with the reference solution’st04, the count is5 of 13 caught. Your count depends on what your test checks. Expect most of the thirteen to stay missed for now, because four short tests aren’t meant to catch thirteen bugs.t03runs a singleORI 5and checksr0. Why is that enough to catchB11, a tier-5 bug?TipAnswerORIis the only way to get a constant into this machine, so a bug on that path is on the path of every program. The tier is how hard a bug is to think of, not how hard it is to trip over. You’ll meetB11again in Lab 06.
Check: you have the check-all output saved.
Expected results
| Check | Expected | |
|---|---|---|
make shuttle final count |
13 of 13 |
exact |
make shuttle wall clock |
about 30 s | ballpark |
make sim, shipped file |
35610.00 ns total |
exact |
The ORI cycle count in the log |
5 clock cycles |
exact |
make waves output file |
waves.vcd, 337,999 bytes |
exact |
Icarus sorry: notes per compile |
28 | exact |
make check, shipped file |
MISSED, VERDICT: FAIL |
exact |
make check, when you are done |
caught, VERDICT: PASS |
|
make check-all, shipped file |
1 of 13 caught |
exact |
make check-all, reference solution |
5 of 13 caught |
exact |
Finishing
You’re done when make check says ok on the golden core, caught on B01 and VERDICT: PASS, with your new test asserting both r1 and the flags.
Submit: nothing for this lab. Keep the check-all output you saved in Part 8.
Going further (optional)
See what Verilator’s build costs and what ccache saves. In labs/common:
make clean && make sim SIM=icarus
make clean && make sim SIM=verilator && ccache -s | grep Hits
make clean && make sim SIM=verilator && ccache -s | grep HitsBoth simulators give TESTS=6 PASS=6 FAIL=0 SKIP=0 and 512010.01 ns, with Verilator’s REAL TIME (the column after the simulated time) about half of Icarus’s.
ccache keeps the compiled C++ files, and its Hits line counts across everything compiled in this container. Each Verilator build compiles six files. If you did Part 5 in this container, its Verilator builds have already filled the cache, so both builds here are fast, and the line reads Hits: 12 / 18, then Hits: 18 / 24. The cache is lost when the container stops, so to see a cold build, run these commands first thing after a restart. The first Verilator build then takes about 16 s end to end, against about 6 s for Icarus, because Verilator builds a C++ program first, and the line reads Hits: 0 / 6; the second takes a few seconds, and the hits go up by six, to Hits: 6 / 12 (50.00%). Two simulators explains the numbers.
Next on the Course path: read Flags as architectural state, then do Lab 03: Three blocks before the processor.