Everdrive SDK / SD card IO

[megatron-uk]

Has anyone made any further progress in interacting with the Everdrive SD filesystem side of things from the PCE?

I found the following posts relating to reading from the SD card:

http://krikzz.com/forum/index.php?topic=69.msg358#msg358
http://pcedev.blockos.org/viewtopic.php?f=5&t=90
https://gist.github.com/BlockoS/3240689 (or saf

Unfortunately it doesn't look like any of the work has got to a point where it is easy to include a library within HuC to read and write from the SD card... though some of the above is very promising. I'd love to be able to write a simple tool to backup/restore save ram to/from the SD card, and with the BRAM functions in HuC it looks fairly trivial to do so (I'm poking about with some test code in Mednafen to read out individual saves - the existing hooks in HuC are really quite nice!).

Sadly my assembly skill is virtually non-existent, so I have no idea where to go from here. It would seem the easiest way forward would be get the edio.c library from Krikzz to a point where reading (and writing!) sectors to and from the SD card is as simple as calling diskReadSector() and diskWriteSector(). Add an embedded FAT wrapper around that (PetitFS has been mentioned in the above posts - 4k library plus 44 bytes work ram) and it should be possible to use the Everdrive as an (effectively) unlimited storage device with normal C-style file() calls.

Info - As of January 2014, the source code for my everdrive-fat HuC library is available on github at:
https://github.com/megatron-uk/everdrive-fat

This also includes the low-level SD library as written by MooZ. I will be trying to keep it as closely synced to my current local version as possible. If I commit any major new functionality I will post the details in this thread.

2014-02-03
Committed latest everdrive-fat code to github. Optimised several math32.h functions and removed debug code from fat-files.h. Initial basic implementation of file read functionality via fread(). We can read file contents now! Still to test multi-cluster directories (don't use with directories with lots of files in as yet!).

2014-01-27
Updated github, added support for fopen(), fclose() in fat-files.h, directory traversal and finding file entries within root or sub directories. Multi-cluster directories not yet tested.

2014-01-21
Initial commit. Added test example which scans your card/everdrive to look for FAT partitions and report FAT filesystem details via fat-dev.h and fat-vol.h. No user-accessible functions available in fat-files.h yet.

[TurboXray]

MooZ is the man

[megatron-uk]

I've been in contact with MooZ and he's updated his ed_sd.asm library for interacting with the Everdrive.

The latest version is available here:
https://gist.github.com/BlockoS/3240689

He's also updated the syntax to be callable from HuC. I've put together a basic set of tests around the asm code:

#asm
   .include "ed_sd.asm"
#endasm
main() {
   char ed_error;
   
   init_screen();
   put_string("init_screen", 1, 0);
   
   put_string("ed_begin", 1, 2);
   ed_begin();
   put_string("OK", 20, 2)   ;

   put_string("disk_init", 1, 3);
   ed_error = 0;
   ed_error = disk_init();
   put_hex(ed_error, 2, 10, 3);
   
   put_string("ed_end", 1, 4);
   ed_end();
   put_string("OK", 20, 4)   ;
   
   for (;;) {
      vsync();
      joytrg(0);
   }
}

Unfortunately I'm currently halting at the disk_init() stage - although things must be partially working as the red led on the everdrive lights up and stays lit as soon as my code runs.

More info when I make any further progress.

[NightWolve]

MooZ is the man

Ah, I remember him from #UtopiaSoft on Efnet IRC. One of the big-time players in the French PCE scene as I recall, or something. ;)

[megatron-uk]

Thanks to MooZ, his latest asm code now initialises the Everdrive hardware and performs a disk_read_open call without locking the hardware. The call to disk_read_sector still returns an error code (62) however.

In the meantime, I'm working on writing a simple FAT filesystem driver in HuC using the docs from:
http://www.pjrc.com/tech/8051/ide/fat32.html

[TurboXray]

MooZ is the man

Ah, I remember him from #UtopiaSoft on Efnet IRC. One of the big-time players in the French PCE scene as I recall, or something. ;)

Yeah. PCE Parisian He's done coding, hacking, translations, demos, etc for PCE.

[megatron-uk]

Raw sector reading is now working courtesy of MooZ. I'm still playing around with things and attempting to read MBR and partition information, but things are looking *very* promising!

[NightWolve]

Don't suppose it's possible to upgrade his loader to add support for a simple save/load state feature ? Just one slot would be great.

[megatron-uk]

Not sure if it would be possible - you'd have to have something PCE-side that was capable of freezing the state of all memory and registers and dumping them to disk. Without running some sort of monitor process on the FPGA to do it, I'm not sure how you'd grab all that state info. I don't know if it's even feasible for the Everdrive side of things to read back from the console hardware - the process is normally the other way around.

My idea was simply to make the existing BRAM functionality built in to the standard PCE hardware more flexible via writing a tool for loading pre-existing saves from SD and being able to save them back out again.

BTW - I'm now in the midst of trying to extract partition table information from the sd card - I'm aiming to use the first available FAT partition of the 4 possible primary partitions for now. The MBR layout is pretty simple:

446 bytes boot code, followed by a 64byte partition table (4 x 16 byte data areas), then by 2 checksum bytes (always 0x55 & 0xAA)

I'm picking up the two check bytes now (at least my mbr is valid, lol!) and am writing some code to extract the filesystem type (byte 5 of the 16 byte area) and LBA start sector (bytes 9 - 12).

Once I have the start sector I can then pass the info to a second disk_read_sector call to read the FAT volume sector for that partition, which contains info about where the filesystem starts and where the root directory is located on disk.

[NightWolve]

I don't know if it's even feasible for the Everdrive side of things to read back from the console hardware - the process is normally the other way around.

Ah yeah, you got a point. Unless this has been accomplished in other flashcarts, can't think of a case ATM, but I wonder if it was done. Has it ?? I dunno... Well, I guess the only feasible cheats are what somebody here was already doing which was to catalog all the offsets in PCE ROM files that correspond to number of lives, or continues, etc. and overwriting them to activate a cheat prior to loading the ROM.

[TurboXray]

If you can write to the SD card from the PCE side, then you can save BRAM quite easily to the card. But save state? Some registers are read only. You'd have to hack the game to keep track of those regs (write hook code to make copies to a set in ram. And extra ram at that, like CD base 64k ram). Else you'd need a pass through card device, that monitors the bus and save those writes (as well as keeps track of start/end of the video frame - to sync with). BRAM yes, save states: per game hack dependent.

[TurboXray]

Oh man, I just thought of something; if you can get write support for the SD card... music tracker time! Both MooZ and I worked on trackers native to the PCE, but there was no way save your songs (other than some convoluted method). That would be awesome, if you could save them to the card. Or even load instruments/samples/songs from the SD card (into ram, like CD base ram). I've already got mouse support code up and running for the PCE mouse.

 Make this happen, megatron-uk!

[esteban]

Oh man, I just thought of something; if you can get write support for the SD card... music tracker time! Both MooZ and I worked on trackers native to the PCE, but there was no way save your songs (other than some convoluted method). That would be awesome, if you could save them to the card. Or even load instruments/samples/songs from the SD card (into ram, like CD base ram). I've already got mouse support code up and running for the PCE mouse.

 Make this happen, megatron-uk!

I would buy a PCE mouse simply for this!

[megatron-uk]

Just been working on a basic set of math functions for 32bit integers in HuC (needed for FAT32 LBA addressing), now that I have that done I should be able to start looking at reading directory and file positions from the volume sector at the beginning of the filesystem.

MooZ has stated he is going to tweak the asm library to tidy things up and then turn his attention to write support.

It might take some time, but I think we have got a good chance of getting to a point where we can use the Everdrive as a general purpose filesystem.

[TurboXray]

It'd also be nice to be able to dump CD data (all of it) directly to the SD card as well.

[SamIAm]

It'd also be nice to be able to dump CD data (all of it) directly to the SD card as well.

You know where that could come in handy? Dumping LD-ROM data tracks.

[megatron-uk]

Here's the latest state of my implementation.

It uses MooZ's Everdrive sd.asm library to initialise and read sectors from the card - my first few functions implement FAT MBR and partition scanning. By default it will return the first FAT partition it detects on the card (but will show any others it finds). I'm part way through reading the filesystem start addresses from the volume sector of the detected FAT partition.

I've also attached the basis of my 32bit (4byte char array) math library needed to do 32bit sector calculations - before anyone says anything, yes, I know the multiplication functions are dreadful (calls add_int32 over and over), and they need to be redone as bitshifting ops.

Anyway, now that that's out of the way. I'd love some feedback from others - does the included test programme detect your Everdrive and SD card OK? Does it pick up your FAT partitions?

Although no write functionality is included and I've not had any filesystem issues so far, I take no responsibility for this code destroying your SD card, setting your dog on fire or releasing bubonic plague in your household.

[megatron-uk]

... and here are the assembly Everdrive functions and the HuC interface.

All credit to MooZ for these two files.

I've attached an image of what I get on my Express with a SD card with one primary FAT partition. It *should* detect and handle up to 4 primary partitions. Extended and non-dos partitioning schemes (GPT, bsd disklabel, etc) are not supported (as the Everdrive doesn't support them anyway).

A quick runthrough the display:

disk_init - does the asm call to the Everdrive registers succeed - if so, what SD card type is detected?
getMBRPart - could we read sector 0 of the SD card?
1be, 1ce, 1de, 1ee - results for reading the first four primary partition records - their filesystem type (FAT32 is 0x0B and 0x0C) and a summary of whether the partition was detected as FAT or Non-FAT
getFATVol - could we read the volume sector from reading the LBA start address of the detected partition?

Hopefully, with a working combination of ED and SD card, you should see something similar.

[BlueBMW]

It'd also be nice to be able to dump CD data (all of it) directly to the SD card as well.

You know where that could come in handy? Dumping LD-ROM data tracks.

A very good idea!  While most people dont really care for laseractive stuff, if its not emulated, eventually it will be lost.  Then five of us will cry

[megatron-uk]

Latest version of my Everdrive FAT test utility - MBR and partition detection code are now very robust, currently working on extracting start sector, data cluster start sector, FAT table offset etc, from the volume sector of the detected partition.

Also updated the test code using these functions to scroll back and forth through the process -> scan mbr, detect partition, load partition volume sector etc, as well as print out a lot of current debug values on the bottom half of the screen.

Feel free to give the current version a try.

(image is code running in mednafen and the error messages shown when no Everdrive is detected).

[megatron-uk]

In the latest version I've refactored some code - removed several local variables no longer needed and have finished the routines to extract FAT filesystem geometry and cluster information from the FAT volume sector.

The next step is to use the information retrieved about the root directory cluster in order to read directory enries and then follow the cluster chains to read file entries and contents.

The latest binary attached should detect any of the 4 primary partitions of an SD or SDHC card, use the first FAT partition detected and extract and display the volume information.

I'm hoping to implement a basic C 'stdio' set of file functions (fopen, fclose, fread, fwrite, fputc, fgets, fseek) on top of these functions to make use of the Everdrive SD interface in a more standard way. Low-level write functionality is currently being worked on by MooZ, so that may come later.

[megatron-uk]

Starting up the tool - screen 1 shows detection of SD card type using the low level sd assembly library.

The second image shows the detection of the four primary partitions and their status as FAT or non-FAT. In this example my 8gb SDHC card has a single primary FAT32 partition.

[megatron-uk]

The third image shows the retrieved address of the volume sector for the currently selected partition (p1 in this example) and the volume sector being read.

The fourth and final image shows FAT filesystem information (no. of sectors per cluster, no. of reserved sectors, size of FAT, start address of data clusters etc). At this point we have the address of the root directory entry and can start reading directory entries and file names.

[megatron-uk]

Oh yeah. Anyone spot the obvious mistake in the last image?

Yes, that's right; the cluster number for the root directory hasn't been switched from little to big-endian yet (it should be cluster 2, not 200 million). Endian issues have gotten me a quite a few times so far!

[NecroPhile]

Obviously. 

I seriously have no idea what's going here (anything past "push run button" is too confusing for me), but I love it.

[Lochlan]

That's awesome that you're doing this, I am very excited!  Thank you so much!

A request:
I greatly (hugely) appreciate you posting source for this--but would you consider throwing it up on github?  Source control is great, and there's also the theoretical possibility of contributors via pull requests.  It seems like a good idea--right?  If you have something against using github there's also bitbucket.

Aside: one of the biggest problems with homebrew console dev IMO is that very few people throw their code into a publicly-accessible remote repository.  Easily-accessible source is just good for other hobbyist devs, and it's also a useful way to preserve these efforts.  Public github repos are likely to outlive forum posts!

[megatron-uk]

I'll certainly look in to doing that. I have a local subversion repository, but it makes sense to have the code publicly available.

[megatron-uk]

I've commited the current code to Github - it lives at:

https://github.com/megatron-uk/everdrive-fat

It's not ready for general use yet however - neither directory or file access are implemented yet, so you can't do anything constructive at the moment.

I'll add documentation on how to use it within the next few days, but the test application I've shown the screenshots of is also included in the repository.

[megatron-uk]

I've moved on to the actual fopen/fclose, fread/fseek functions now - things you can actually use in your code to interact with the FAT32 filesystem.
 
So far I have directory traversal working for my fopen() implementation - for now my design decision is to focus on the classic 8+3 filename support as LFN support require a lot more work (and work memory - twice the storage for the same length of filename). I can't see that being a major problem in the short term - all LFN entries have a corresponding 8+3 representation anyway.

Anyway, given a normal file path (eg "/files/saves/soldier.sav")* I'm able to traverse the root and then sub-directories to find the file and record a free 'file pointer' against it (really just an index into an area of memory i'm calling the file work area that holds the directory entry for a file, its starting cluster, current cluster and current seek position within the file - struct support would really simplify things here).

The maximum number of open files is configurable at compile-time - and the size of the file work ram is a product the number of files you want to be able to support at any one time (minimum of 2 - one for internal directory operations and 1 for user files) multiplied by the number of bytes neded to store the metadata for a file (currently 44 bytes). So a minimum of 88 bytes.

When I've got the last kinks of fopen() worked out I'll commit the changes back up to github.

* Incidentally, the code supports both Unix-style ('/') and DOS-style ('\') directory name seperators. eg:

Code Select Expand

fopen("/files/data/cheats.txt")
and

Code Select Expand

fopen("\\files\\data\\cheats.txt")
are equivalent (backslashes must be escaped in HuC otherwise the compiler chokes). Paths are always relative to the current active partition, as selected with getMBR(part_number) or getMBR(0) to select the first active FAT parition.

[TurboXray]

Write operations are going to be next on the list?

[megatron-uk]

MooZ is working on the low level write functions I believe. If he can do something like his read implementation then hooking it in to a fwrite() call should be fairly straightforward as the hardware reads and writes single sectors at a time.

I need to do some speed tests shortly - I'm at a stage where I can do some filesystem-level read tests now.

[megatron-uk]

Committed the latest code to github. Has support for fopen() and fclose() - errors are stored in 'everdrive_errors' global var:

Code Select Expand

char filename[256];
strncpy(filename, "/path/to/a/file.txt", 19);
f = fopen(filename);
if (f != 0){
    put_string("opened file ptr", 0, 1);
    put_number(f, 2, 16, 1);
    fclose(f);
} else {
    put_string("error opening file", 0, 1);
    put_number(everdrive_error, 3, 19, 1);
}
Next to do is fread() and fgetc() and properly test fopen() with directories that span multiple clusters (my test setup has a rather small directory that fits inside a single cluster - one sub folder and two files).

[megatron-uk]

Anyone got any tips for timing function calls? I'm doing some basic rough and ready approximations of the transfer speed I'm getting with the Turbo Everdrive (64-100kb/sec by my estimations), but I'm not sure my use of clock_tt() is particulary accurate (or indeed useful with longer running tests, ie >512kb).

[megatron-uk]

It is indeed very interesting - as BonkNuts mentioned in another thread, the CD ram (64k) and SuperCD ram (192k) are also still available while the Everdrive is inserted (albeit under certain conditions) so we seem to have quite a bit of flexibility.

The ultimate goal for myself is indeed to have a utility that can read/write BRAM to/from the SD card - write support may be a little longer - but I really don't think I'm too far away from being able to read save files from the SD card (created in an emulator perhaps) and loading it into BRAM on the system. Saving records back out will be the icing on the cake!

[megatron-uk]

Ok, I played around with the clock_xx() functions a bit more and have got some results that are repeatable, so I'm a bit happier now. Bear in mind that these figures are only accurate to 1/6 of a second.

Test: 512byte sector from SD card
Time: 0 s, 0 ticks -  less than one vsync interval or clock 'tick'.

Test: 64kbyte read from SD card
Time: 0 sec 37 ticks

Test: 128kbyte read from SD card
Time: 1 sec, 15 ticks

Test: 256kbyte read from SD card
Time: 2 sec, 30 ticks

Test: 512kbyte read from SD card
Time: 4s, 59 ticks (lets say 5 sec)

I work that out to be a transfer rate of approximately 115kbytes/second. That's really not bad at all. I would think that with some optimisation we could get pretty damn close to the speed of the CDROM drive, with, of course, a hell of a lot lower latency in access time.

Notes:

• All tests where done after a call to fopen() - so the time to seek to a file and find it's directory entry were not considered (will be dependent on directory hierarchy).
• The test was fairly dumb - just sequential sector reads, I'm not even paying any notice of where the file ends yet.
• I'm recalculating LBA address from the file cluster number on each successive 512byte read, my 32bit math multiplication function, well, isn't multiplication at all, it's a massive huge hack in the form of an add, so there are definite savings to be made there.
• A cluster <-> LBA sector lookup for the most recently accessed file entries would be worthwhile looking at too - it depends on how often you reopen the same file or indeed how slow that is (next test!).
• There will be some overhead in retrieving clock_tt(), clock_ss() etc, at the end of each test, but the speed seems to be consistent between the different transfer tests so I would imagine it's a very small, consistent error number.

[Lochlan]

Have you considered using GNU Make for your project's build system?  Rather than having a bunch of shell scripts you could wrangle everything into one Makefile with multiple targets.  It would also be nice to have access to make clean!  I don't know if you like GNU Make, but I use it to build all my programming projects, commercial and personal.

[megatron-uk]

Yeah, I probably should do. Once I get the fix to the 32bit math functions in place and fread() working correctly again I'll take a look at it.

[Psycho Punch]

I work that out to be a transfer rate of approximately 115kbytes/second. That's really not bad at all. I would think that with some optimisation we could get pretty damn close to the speed of the CDROM drive, with, of course, a hell of a lot lower latency in access time.

Now I only wonder if a CDROM emulator via HuCard door is possible...

[megatron-uk]

Latest commit to github has fixed the math32 mul_int32_int8 function, removed some superfluous debug code in fat-files and generally cleaned stuff up quite a bit.

The fread() function now works correctly (correct cluster and sector address is used!), but is not yet tied into any logic for reading multiple sectors, incrementing file position pointers after a read etc. i.e. it will read the first sector of a file each time you call it.

All those provisos being true, this is still a major milestone in the project.

We can read files from the SD card from our running PC-Engine games and utilities!

I attache the output from a simple test routine (examples/test-files.c in github) that scans the FAT for a file called "/text/dracula.txt" using fopen(), displays directory entry info for the file and then makes a call to fread() to read the first 512 bytes and display them on the screen (I'm not formatting anything here, just demonstrating the ability to read and display data from a file on the filesystem).

One more thing - I would really appreciate feedback from anyone else who tries the code out; specifically the examples/test-speed.c programme... I'm getting some awfully quick results (at least twice as fast as I reported previously) and I don't know if I should trust them yet.

[megatron-uk]

First several hundred bytes of the eBook 'Dracula' from Project Gutenburg:

[TurboXray]

Awesome work! I'll do some tests later this week, if you need.

[megatron-uk]

That would be really helpful - just as long as I don't find that this code doesn't work on anything but the two micro-sd cards I've been testing it with!

[megatron-uk]

Well my application and interview for a new job went well (got an offer and accepted it) so I'm going to try to find some time this weekend (after taking the wife out for dinner for valentines, of course!) to get the fread() implementation finished (so that sequential sectors of a file can be read) and fseek() started (so that position within a file can be changed).

[pceslayer]

any progress?

[Lochlan]

any progress?

If you looked in the Cyber Knight thread (next to this one), where megatron is also active, you'll note that he is busy with a move.

And IDK if he feels this way, but IMO it's pretty rude to ask for status updates like this unless it's been like several months.  You'll get your free software when you get it.  I definitely understand the anticipation, though, I'm looking forward to this release as well!

[megatron-uk]

Haven't done much for the past two weeks - since the last update I've been working on file position seeking - ie, read all the sectors in a cluster for a file, lookup the next cluster from the FAT table. Things are still moving, albeit slowly, due to my simultaneous work on the Cyber Knight translation and, as Lochlan said, my wife and I have just sold our house, so virtually all of my electronics and games are now packed up or in storage. I can still develop on my laptop of course, but it will be difficult to test anything on real hardware for the next month or so.

Don't worry, it's my intention to get at least full file read support working. You can of course use it to read file data from the SD card now, even with the functions that are already there. Just seeking (and thus FAT traversal) needs to be finished for full read support via fread().

[pceslayer]

any progress?

If you looked in the Cyber Knight thread (next to this one), where megatron is also active, you'll note that he is busy with a move.

And IDK if he feels this way, but IMO it's pretty rude to ask for status updates like this unless it's been like several months.  You'll get your free software when you get it.  I definitely understand the anticipation, though, I'm looking forward to this release as well!

I stumbled across Mooz original thread on a different forum and saw megatron-uk had picked it up and ran with it on here.

So I asked... Didn't mean to be rude... sorry

[MooZ]

Hello there!

I'll resume working on write operation anytime soon. I got busy with demo stuffs...

[megatron-uk]

Hey!

Long time no see 

I've not done much for a good few weeks now either - got sidetracked by Cyber Knight translation and the fact that all my toys are packed up and in storage in anticipation of moving now!

Hope to get some more work done on the API side of things for HuC when things calm down a bit! 

[MooZ]

First things first...HuC complained about some text in the comments. In the end it was a newline issue. I had to add the byte flag to fopen calls in io.c. I can post a patch for the curious ones.

Anyway! It seems that something is missing. pceas complains about an undefined routine called _mem.
Maybe you forgot to add some file? It's called in fat-files-extras.h.
By the way, it feels that something have been eaten from line 432.