Eternal Sunshine of the Spotless Code

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Andrew Davie

AI suggested the title. Eternal Sunshine happens to be one of my favourite movies.

Anyway, it's in reference to me being quiet for a few days. I've been hunting a rather obscure bug. In fact I had to call in the big guns and asked @JetSetIlly to help me out. He found the issue in short order; I was completely stuck.

The symptoms were that running in Gopher, the game would randomly crash every now and then. No real clues that I could see as to why - and the crash changed when I changed pretty unrelated things in the code. I could take out just about everything and it would still crash. Stumped!

So, now's a great time to talk about how the CDFJ code/bank-switching works. Please correct me if wrong, those who know better... i'm really just a novice user and have formed my own guesses as to operation of things.

As I understand things, the ARM on the cartridge is monitoring the bus - and when the 6507 puts an address on the bus, then the ARM puts the contents of that address on the bus.  Something like that.  Now the 6507 if blissfully unaware it's "talking" to an ARM processor and not actually to memory - so it behaves normally.

In a '2600 game we have a bit of spare time to "do stuff" in both the vertical blank and the overscan areas of a TV frame. Usually, the 6507 sets and waits for a timer to get those blanking areas the correct number of scanlines. This means that the 6507 is (effectively) idle, although many assembler games do gobs of stuff during that time.

With CDFJ games, these blanking areas are an opportunity for the programmer to say to the ARM CPU "hey!! there's gobs of time here, the 6507 is just idle, go and do some complex stuff for a while".  That's all well and good, but although the 6507 is "idle", it's not *stopped*.  It is continually running and fetching instruction after instruction, and increasing the program counter as it gets each new one.  The way that CDFJ works is that the 6507 signals to the ARM that it's going to be idle - and the ARM chip puts a NOP on the bus and then just ignores the 6507 requests for data from any/all addresses. As far as the 6507 is concerned, it's running a very long program consisting solely of NOP after NOP after NOP, consecutivei in memory.  That is, the 6507 fetches an instruction from address. It sees a NOP, so it executes that and increments the program counter. The it fetches an instruction from the next address (which the program counter points to)... ad infinitum.  Essentially the 6507 is "idle" but not really. It's just executing lots of NOPs becuase NOP is "stuck" on the bus data no matter what address the PC is at.

ALMOST. Unfortunately, once the PC gets past the end of the program address range (i.e., 4K), the program counter points to low memory - which includes RAM and the hardware registers. The hardware, when asked for THOSE addresses, does NOT fetch the data from the bus (that we were accessing before).  So instead of seeing NOP, we see (effectively) random data. But the 6507 tries to execute that as if it were valid code. So just about ANYTHING could happen. We could jump somewhere, or try to return from a subroutine, or have an unimplemented/illegal opcode. Anything.

There's no way for the ARM to detect/prevent this, because it is doing all those complex calculations that the 6507 gave it permission to do. "Hey, i'm idle! Do your stuff!!!"  remember?  So this puts the onus of the programmer to do the following...

1) Make sure the 6507 PC is very very early in the address range when you start your "idle" with the NOPs as described above. This maximises the time before the PC ever gets to the end of the address range and wraps around (basically, approaching 4K of NOPs --> 8K of 6507 cycles).

2) That in turn defines the maximum number of cycles you have to do stuff, before the ARM *MUST* restart servicing the bus and providing valid data for the 6507.  So, code on the ARM should be aware of the maximum cycles it can use.

Now there's a very convenient timer on the ARM that you can check to see how many cycles you have used. So the crux of the ARM side of things is to kinda know how long something might take to do (maximum) and only do it if there are sufficient remaining cycles in the allocated time. It's simple enough.

Now to my bug:  I was wanting to consolidate space; the C code lives low in the cartridge memory, and the 6507 code is at the top. I had 4K reserved for the 6507 code (i.e., it started at about the base of the address range allocated to it).  $7000-$7FFF for those interested.  And my idle loop (as above) triggered the NOP thingy at about $7010; very early.

I forgot ALL OF THIS when I decided to consolidate space. I just calculated exactly how much 6507 code I was using, and shifted it all up as high as possible (to free space for more C cod).  The idle loop ended up at about $7C10.  And everything worked just as I expected.

Only, it didn't.

Some time later (way after I did this change, and so unrelated in my brain)... Gopher started crashing when running my game. I didn't associate it with the NOP waits, only @JetSetIlly caught that. Lucky for me, too - I was stumped.

Now let's look at some timings.  My vertical blank is about 32 scanlines - give or take.  At 76 cycles per line that's 32 * 76 cycles (2432 cycles total).  Divide by 2 to get the number of NOPs that are executed in that time (-->1216 NOPs). OK, fine. If all is working properly, and we enter VB and tell the  ARM we're idle, then we need the ARM to return servicing the BUS before 2432 6507 cycles.

Also, we need to ensure that our idle "loop" starts at least 1216 bytes before the end of the address range. That is, around $7FFF-1216 (=$7B40).  As it happens, my code squishing - shifting the 6507 code to the top of memory actually ended up putting the NOP idle loop start at $7C10 give or take.

What's the consequence of that?  Well, it means that rather than having 1216 NOPs before we run out of address space, we actually only have around 1000.  But the ARM code doesn't know that. It just knows how many cycles are in 32 scanlines (it's 70000000/262/60 == 4452 per scanline). Usually, mostly, nearly always, my ARM code finished its stuff well before that -- I had my "available time" set to something like 130000 cycles.  BUT, but... but.... every now and then, because of the dynamic allocation of tasks, the ARM code managed to fit things really really neatly and was able to maximise the stuff it would do in the available time it thought it had.

Meanwhile, because the 6507 idle loop was shifted up in memory, it was chugging along with the PC getting way high, and eventually wrapping past $7FFF (back to 0). And *bang* we're in random la-la land - gopher goes black-screen on me. No real correlation to anything in particular happening in the game, because different things were happening each and every crash. I could take things out, but other things were still running in that dynamic allocation of stuff to do on the ARM side. All that was required was for the scheduling to be efficient and jam-pack stuff into the available time properly and every now and then we hit a spike and overflow a certain waypoint (1000 bytes of NOPs = 2000 cycles of 6507) and we're a dead duck.

So, to test all this, I did some calcuations and figured that putting the NOP idle loop at $7B00 (instead of the $7C10) gave sufficient NOP "leeway". In fact I could have put it at $7B40 but I see little need/reason to tempt fate with my calculations. Since making that change everthing is, as expected, rock solid once again.

Huge thanks once again to @JetSetIlly -- I hope that Gopher can catch this sort of really hard-to-find bug at some future point.  As an aside, it did not show up in Stella because - I think - ARM instructions are not executed synchronously with the 6507?  Anyway, it would not have worked properly on actual hardware, so Gopher was reliably emulating exactly what the hardware would do.

Phew; very glad to have that one behind me! Random crashes are never fun.

SpiceWare

#1
Quote from: Andrew Davie on 23 Oct 2025, 02:19 PMPlease correct me if wrong, those who know better...
You got it! I'm impressed you have that much free space in the single 6502 bank. One thing you can do is fill up the middle of the 6502 bank with data, the ARM can still access it, which lets you have the NOPs start at the beginning of the bank.

Quote from: Andrew Davie on 23 Oct 2025, 02:19 PMAs an aside, it did not show up in Stella because - I think - ARM instructions are not executed synchronously with the 6507?

Unless it's changed (I haven't kept up) as far as Stella is concerned any ARM code takes 0 cycles of 6507 time. In 2017 I added support for ARM Timer 1 to Stella so my console detection routines (for NTSC, PAL, and SECAM) would work on it. It works by telling the ARM emulation how many 6502 cycles have passed since the ARM routines were last called. The ARM emulation multiplies the 6502 cycles by the correct value for the console (NTSC/PAL/SECAM) that Stella is emulating:

  constexpr double NTSC   = 70.0 / 1.193182;  // NTSC  6507 clock rate
  constexpr double PAL    = 70.0 / 1.182298;  // PAL   6507 clock rate
  constexpr double SECAM  = 70.0 / 1.187500;  // SECAM 6507 clock rate

Following images linked to my post on March 15, 2017. You might need to view the post at AA for the linked images to show up here.

NTSC:


PAL


SECAM:






Andrew Davie

Quote from: SpiceWare on 24 Oct 2025, 02:12 AMYou got it! I'm impressed you have that much free space in the single 6502 bank. One thing you can do is fill up the middle of the 6502 bank with data, the ARM can still access it, which lets you have the NOPs start at the beginning of the bank.

Yes, I did that in most earlier CDFJ stuff I did. Learned, of course, from your collect3 demo I think it was called. Thanks, in retrospect, for that. But I ended up rather NOT liking that technique; I recall your comments saying you preferred to put data there but it ended up annoying me more than anything; I preferred C structures and putting everything in C.  Given, now, I've calculated I only need to shift the NOP loop to no later than $7B40, I don't see much need to do anything else.

QuoteUnless it's changed (I haven't kept up) as far as Stella is concerned any ARM code takes 0 cycles of 6507 time. In 2017 I added support for ARM Timer 1 to Stella so my console detection routines (for NTSC, PAL, and SECAM) would work on it.

Seems like a good addition that should be included. I don't use Stella at all these days, except maybe for confirmation that a bug I'm seeing is not a Gopher thing. Gopher for me offers by far the better developer experience. It's a remarkable emulator and I can usually get fixes/changes from @JetSetIlly pretty quickly when required.

JetSetIlly

#3
Quote from: SpiceWare on 24 Oct 2025, 02:12 AMUnless it's changed (I haven't kept up) as far as Stella is concerned any ARM code takes 0 cycles of 6507 time. In 2017 I
There's now a developer option for Stella so that it accumulates cycles according to the ARM instructions executed. When CALLFN finishes executing the screen position should be nearer the correct place. What Stella doesn't seem to do (yet) is to go through the process of fetching the next instruction according to the address bus while CALLFN is active. There seems to be an assumption that if CALLFN is active then the next instruction is a NOP.

In other words, cycle accumulation is only half of the solution in this particular case.

For Gopher2600, I do go through the complete fetch process. Meaning that if the program counter runs over into non-cartridge space, it starts to execute all sorts of instructions, most likely ending in a BRK and a KIL

That said, the way Stella does it is a great optimisation. There's no doubt Gopher2600 loses some speed because of the additional fetches. The downside of the optimisation is that it assumes the ROM is "correct".

Quote from: SpiceWare on 24 Oct 2025, 02:12 AMIn 2017 I added support for ARM Timer 1 to Stella so my console detection routines (for NTSC, PAL, and SECAM) would work on it. It works by telling the ARM emulation how many 6502 cycles have passed since the ARM routines were last called. The ARM emulation multiplies the 6502 cycles by the correct value for the console (NTSC/PAL/SECAM) that Stella is emulating

That was superb work. It's a very clever use of the ARM. If we used the ARM for nothing else except providing a benchmark for timing it would be worth having.
https://github.com/JetSetIlly/Gopher2600
@JetSetIlly@mastodon.gamedev.place
@jetsetilly.bsky.social

SpiceWare

Quote from: Andrew Davie on 25 Oct 2025, 03:33 PMI recall your comments saying you preferred to put data there but it ended up annoying me more than anything; I preferred C structures and putting everything in C.

Not so much that I preferred it, but that I found the space savings to be significant enough to be worth the hassle.

With the data in 6502 code the table entries were 2 bytes, all that's need for a 32KB ROM.

When I attempted to do it all in C the tables doubled in size to 4 bytes per entry. I tried a number of things to just store the lower 2 bytes of the pointers, but could never get it to work.

With the larger ROM options of CDFJ+, up to 512KB, I was planning to put all the data in C.

SpiceWare

Quote from: JetSetIlly on 25 Oct 2025, 04:59 PMThat was superb work. It's a very clever use of the ARM. If we used the ARM for nothing else except providing a benchmark for timing it would be worth having.

Thanks, that means a lot!

Andrew Davie

Quote from: SpiceWare on 28 Oct 2025, 01:44 AMWhen I attempted to do it all in C the tables doubled in size to 4 bytes per entry. I tried a number of things to just store the lower 2 bytes of the pointers, but could never get it to work.

Nor could I, and I spent many frustrating hours/attempts trying to do so.
But I've had another go and have now cracked it!

// in header....
extern uint16_t frameLowTable[];


// table definition...

__asm__(".section .rodata\n"
        ".global frameLowTable\n"
        "frameLowTable:\n"
        ".hword climb_3_6\n"
        ".hword blue_0_0\n"
        ".hword blue_0_0\n");

// c usage...

    unsigned char *fp = (unsigned char *)(frameLowTable[ch]);
    unsigned int mask = fp[trix];

Andrew Davie

Quote from: Andrew Davie on 28 Oct 2025, 12:39 PMBut I've had another go and have now cracked it!

One caveat; the reference to the data in the __asm__ table is not seen by the linker WRT to optimising out un-referenced data. So the frame gets discarded unless its referenced elsewhere, and then the __asm__ will fail with an unreferenced symbol. I'm working on this.

Andrew Davie

Quote from: Andrew Davie on 28 Oct 2025, 02:15 PM
Quote from: Andrew Davie on 28 Oct 2025, 12:39 PMBut I've had another go and have now cracked it!

One caveat; the reference to the data in the __asm__ table is not seen by the linker WRT to optimising out un-referenced data. So the frame gets discarded unless its referenced elsewhere, and then the __asm__ will fail with an unreferenced symbol. I'm working on this.

Solved...

__attribute__((used)) const unsigned char blue_0_0[] = {

Place the __atribute__((used)) in front of C structures referenced in the __asm__ block and not elsewhere in the C code. Linker will not discard them.